Running FreeBSD / OpenBSD / NetBSD as a virtualised guest on Online.net

I’ve been running a mixture of FreeBSD / OpenBSD & NetBSD as guests on a dedicated server at Online.net. While getting the operating systems installed was fairly seamless, getting networking going was not.

  1. Client are not isolated in a layer 2 domain
  2. DHCPv6 config is broken

Clients not being isolated is not so much a problem itself and is typically what you’d expect if you plugged a bunch of computers into a switch with a single VLAN or unmanaged switched for example; but in a shared environment with untrusted tenants it can cause problems. Broadcast & IPv6 multicast floods aside, one is open to most of the attacks in something like THC-IPv6 due to lack of MLD snooping which would prevent a rogue IPv6 router.

Attacks via IPv6 are not so much of a problem as their use of non-RFC complaint timers settings in their DHCPv6 make it unfeasible to use the offered native IPv6 connectivity as clients will fail to renew leases. Depending on the DHCPv6 client used, the amount of time it takes fail to renew a lease will vary. dhcpcd for example now warns if detects a lease is not compliant with RFC 3315 section 22.4 “Identity Association for Non-temporary Addresses Option”.

Despite having a vast address range in IPv6 and a /48 subnet is allotted free of charge, you’ll need the equal amount of v4 address addresses as the v6 addresses you intend to use at Online.net. There is a way of using a /48 and allocating addresses yourself but it’s only possible using a version of Proxmox which they provide.

You can save yourself a lot of hassle both with configuration & trying to deal with their support  regarding IPv6 by using a Hurricane Electric tunnel. I actually found connectivity was also faster from Hurricane Electric than using the native connectivity.

For IPv4 connectivity on a guest (assuming you’re renting individual IP addresses & not a /27 prefix), you’ll need to use the default gateway IP address assigned to your host alongside the allotted IP address and a /32 prefix.

Assuming the network details are as follows
Default gateway on host: 192.0.2.1
Failover IP #1: 198.51.100.10, assigned to MAC address 00:50:56:00:01:AA
Failover IP #2: 203.0.113.11, assigned to MAC address 00:50:56:00:02:BB
Failover IP #3: 203.0.113.100, assigned to MAC address 00:50:56:00:03:CC

The MAC addresses need to be assigned to the tap(4) interface on the host.
If you’re using bhyve and your guest is using the interface tap0, this would be performed using the -s flag to configure the virtual PCI ethernet card, eg -s 1:0,virtio-net,tap0,mac=00:50:56:00:01:AA

It’s then onto configuring each OS to handle a gateway which is in a another subnet for IPv4 connectivity.

FreeBSD

In FreeBSD you need to construct a route to reach the default IP address first, before you specify the default IP address, otherwise things will not work. So assuming we’re going to use Failover IP #1, your configuration in /etc/rc.conf would be as follows

ifconfig_vtnet0="inet 198.51.100.10/32"
gateway_if="vtnet0"
gateway_ip="192.0.2.1"
static_routes="gateway default"
route_gateway="-host $gateway_ip -interface $gateway_if"
route_default="default $gateway_ip"

Note, the installer at present prevents network installs, you should use a iso image containing the distfiles, bug 206355 has more details.

NetBSD

On NetBSD, configure networking using /etc/netstart.local, entering the commands you’d enter at the console inside the file. Assuming failover IP #2 is going to be used for the NetBSD VM, the following would configure the guest to reach the outside world using 192.0.2.1, as discussed in the NetBSD Network FAQ

ifconfig vioif0 203.0.113.11/32
route add -net 192.0.2.1 -link -cloning -iface vioif0
route add default -ifa 203.0.113.11 192.0.2.1

OpenBSD

On OpenBSD, configure the networking from the ethernet interfaces configuration file hostname.if(5).

Assuming failover IP #3 is going to be used for the OpenBSD VM, the following will setup networking.

/etc/hostname.vio0

inet 203.0.113.100 255.255.255.255 NONE
!/sbin/route add -net 192.0.2.1 -netmask 255.255.255.255 -link -cloning -iface vio0
!/sbin/route add default -ifa 203.0.113.100 192.0.2.1

It’s also possible to not specify the -cloning flag but a patch is required if you’re running 5.9 release.

Hipster keyboard layout on Windows

Windows supports the Dvorak keyboard layout natively, out of the box, so there is no tinkering required outside of visiting control panel & selecting the desired layout.

To switch the location of the control & caps locks keys however, you need to modify the registry & and reboot. I’ve uploaded a registry snippet which can be applied (taken from Windows 7). It implements the changes covered in a post on kodiva.com.

Hipster keyboard layout on NetBSD

Each of the major BSD’s have a different way of handling keyboard layouts on the console & X11. On OpenBSD X11 inherits the setting from wscons by default, on FreeBSD the console keyboard config is separate to the X11 config & depending on if you go down the hald route or not, you may find yourself writing XML to configure your keyboard. For NetBSD which I’ll cover here, wscons configuration is again separate from X11 configuration but everything is configured as per usual via the xorg.conf keyboard layout.

The snippet below is from xorg.conf which sets the keyboard model as a ThinkPad T60 (it should apply to X60 series apart from issues with media buttons), US Dvorak layout with the crtl & caps locks switched.
Section "InputDevice"
Identifier "Keyboard0"
Driver "kbd"
Option "XkbRules" "xorg"
Option "XkbModel" "thinkpad60"
Option "XkbLayout" "us"
Option "XkbVariant" "dvorak"
Option "XkbOptions" "ctrl:nocaps"
EndSection

I didn’t know about the ctrl:nocaps option and I happen to stumble across it in the X section of the NetBSD guide.

To apply the same layout to the console, edit /etc/wscons.conf and set encoding to us.dvorak.swapctrlcaps followed /etc/rc.d/wscons restart.

Not sure how hipster this all is, managed to get sidetracked into NetBSD desktop config as I was working on updating a package in pkgsrc and remembered the tweet above. Seems like a common thing in the emacs world.

Connecting to shared calendars on ical server using an iPhone

To access the shared resource/location calendar on a iCal server from an iPhone via the caldav protocol you’ll need the GUID of the shared calendar.

First, find the GUID of calendar on the server using the calendarserver_manage_principals command, eg
sudo calendarserver_manage_principals --search shared
1 matches found:

your shared cal (Resource)
GUID: 6x3331a8-as12-ea2x-4ou1-ndeb3ct4wa686
Record name(s): 6x3331a8-as12-ea2x-4ou1-ndeb3ct4wa686

Then on the iPhone (assuming the device is unmanaged), create a new CalDAV calendar account & fill in the correct server, username & password field & hit next.
Assuming everything went ok, you should be back on the “Mail, Contacts, Calendars” page, select the new calendar you just created & advanced settings.
Edit the “Account URL”, replacing the UID on the end of the url with the GUID of the shared calendar you noted down on the first step.
eg:
https://username@yourserver.somewhere.local:8443/principals/__uids__/6x3331a8-as12-ea2x-4ou1-ndeb3ct4wa686

CUPS generates “Internal server error” when visiting administration page

It seems that Mac OS Lion server (10.7.0) may have been released with a broken CUPS configuration. Visiting http://localhost:631/admin would generate an “Internal server error”, this seems to be addressed in newer builds (my Macbook Pro which was install from a 10.7.2 image does not exhibit the problem yet a customers server which came bundled with 10.7.0 (upgraded to 10.7.1 > 10.7.2) does).
To resolve the issue sudo cupsctl --debug-logging
Then switch off the debug logging with sudo cupsctl --no-debug-logging

I suspect the problem is due to a missing directory because comparing the default, previous & current revisions of the /etc/cups/cupsd.conf did not show anything that could cause problems.
I’d also go far as to say that you don’t need to turn on debug logging in-order to resolve this issue, it can be any of the options that can be set via cupsctl as it checks the configuration of the system, this is just a theory as I was unable to dig further because I was on a production system. Resetting the printing system did not solve the issue when attempted previously.

Building & administering jails on FreeBSD, Part 1

22/05/2014
These instructions are now part of the FreeBSD handbook since docs/189901 was committed. Please refer to the instructions in the handbook.

The FreeBSD jail(8) manpage & Chapter 15 of the FreeBSD handbook do a great job of explaining jails & helping you get on your way with creating jails, this post builds on that information, covering alternative methods for getting your jails installed & adding what’s not covered already such as maintenance of jails (patching to be specific) & version upgrades.

  • Part 1 (this post :)) will cover alternative install methods & jail maintenance
  • Part 2 (not yet published) will cover upgrading to a new version FreeBSD

Once completed the information from these posts will be submitted for inclusion in the handbook.

So lets begin, when creating a “complete” jail you have two options for the source of the userland, compile from source code or use the prebuilt binaries from install media, both the jail manpage & handbook cover building from source code, we wont go over it again here.

One thing worth mentioning though is if you want to build from source code, create a src.conf file & disable items which are not required, this should speed up the time required to build world & reduce the amount of disk space used by jails.

Here are two sample src.conf files, which disable building items such as firewalls (no use unless you’re using vimage), acpi or documentation:
Sample src.conf #1
Sample src.conf #2

To install the userland from installation media
first create the root directory for the jail, eg
mkdir -p /usr/jails/mynewjail
set the $DESTDIR variable to this location
if using sh
export DESTDIR=/usr/jails/mynewjail
if using csh/tcsh
setenv DESTDIR /usr/jails/mynewjail
mount the media (using the 8.0-RELEASE cd 1 iso in this example)
mount -t cd9660 /dev/`mdconfig -f /some/path/to/8.0-RELEASE-i386-disc1.iso` /mnt

Extract the binaries from the tar balls on the install media into your declared destination, realistically, you’ll only need to extract base, but you can do a complete install if you wish to.
To install just base:
cd /mnt/8.0-RELEASE/base; ./install.sh

You are about to extract the base distribution into /usr/jails/mynewjail – are you SURE
you want to do this over your installed system (y/n)?

To install everything but kernel:
if using sh
cd /mnt/8.0-RELEASE; for dir in base catpages dict doc games info manpages ports; do (cd $dir ; ./install.sh) ; done
if using csh/tcsh
foreach dir ( base catpages dict doc games info manpages ports )

cd /mnt/8.0-RELEASE/$dir; ./install.sh

end

All configuration steps from here on to get up and running are as specified in the jail man page & handbook.

Keeping jails up to date with patches
On a host with default settings the freebsd-update(8) tool doesn’t work as
chflags(1) is not permitted in a jail, set sysctl security.jail.chflags_allowed to 1 to allow it & freebsd-update can be used.
The other option is to patch the userland manually from the host OS. All the needs to be done is the $DESTDIR has to be passed to the make install command eg.
In section 2b of the FreeBSD-SA-10:04.jail advisory you’re told to
# make obj && make depend && make && make install
after patching, instead you would issue
# make obj && make depend && make && make install DESTDIR=/usr/jails/mynewjail

22/05/2014

Use the -b flag for freebsd-update from the host to update jails instead of taking drastic measures.

Configuring OpenSolaris with IPv6 connectivity

To configure OpenSolaris to use IPv6 NDP (neighbour discovery protocol) create an empty file named in the following convention:
/etc/hostname6.interface#:#
first hash being the interface number & the second being a user defined number for a logical interface
eg
/etc/hostname6.e1000g0:1

If you’re having DNS resolution issues, do
cp /etc/nsswitch.dns /etc/nsswitch.conf

To configure OpenSolaris to use a static IPv6 address
create a file using the same convention as mentioned during the NDP stage above & inside it add
addif ipv6address/mask up
eg
addif 2a01:300:200::1/64 up

To configure your default IPv6 router on OpenSolaris
create a file named /etc/defaultrouter6 & add the ip address inside

The instructions above make the changes persist across reboots, if you’d like to make changes to a current session, the configuring an IPv6 network section of the IP services Solaris administration guide is a handy reference.
These instructions should also apply to Solaris as well though I haven’t tested it.
The source of information for this article was the IPv6hostsolaris wiki article.

Running rndc without specifying a port on Mac OS X

The stock BIND config on Mac OS X (both client & server versions) is set to listen on TCP port 54 for control commands via rndc(8), the rndc utility however doesn’t have a config file, so it defaults to using TCP port 953, this mean it has to be invoked with -p 54 to work properly, to rectify the issue create /etc/rndc.conf & add the following to it
include "/etc/rndc.key";
options {
default-key "rndc-key";
default-server 127.0.0.1;
default-port 54;
};

Going IPv6

Looking for something to do this weekend I remembered that a few years back after reading Secure Architectures with OpenBSD I’d signed up to freenet6, I retrieved my forgotten account details & logged in to the portal to get started, then remembered why I never got off the ground with this project, the client is a PITA to setup, a quick search of the ports tree showed up net/aiccu.
The description of the port:
AICCU makes it very easy for anybody to get IPv6 connectivity
everywhere they want. It uses the TIC (Tunnel Information & Control)
protocol to request the information needed to setup a tunnel through
which the connectivity is created.

AICCU supports the following tunneling protocols:
– 6in4 static (RFC 2893)
– 6in4 heartbeat (RFC 2893 + draft-massar-v6ops-heartbeat)
– tinc (http://www.tinc-vpn.org)
– AYIYA (draft-massar-v6ops-ayiya)

As AYIYA even works from behind NAT’s, thus unless there is a very
restrictive firewall in place, anybody should be able to get IPv6
connectivity without problems and everywhere they want.

One does need a SixXS account and at least a tunnel. These
can be freely & gratis requested from the SixXS website.

Before installing the port I headed over to the SixXS website to have a nose around, the FAQ is well worth a read, it lays down the basics of the signup process quiet well.
The only hold up between signing up & getting going with your 1st tunnel is the wait for approval by the project admins which though the site says it can be as long as a week took, it only a few hours for me (even on a saturday!:))
The project works on a credit basis (there is no money inolved), costs are broken down in the FAQ, basically you start off with 25 credits, which allows you to get a tunnel for a single host setup at the cost of 15 credits.
Once you’ve been able to demonstrate that you can keep the tunnel up for seven days you’ll be credited 5 credits, this will give you enough credits to apply for a subnet.
Fingers crossed, if it all goes well I’ll be migrating my home network to IPv6 sometime next week.

The setup of the aiccu client was pretty straightforward, I just needed to provide my username & password to the config file. You don’t actually have to use the aiccu client, you can create a gif(4) manually, this is covered in the faq aswell, but seeing as I’m a n00b at this I will ditch the client once I’m more comfortable. Before you fire up the client you need to configure your firewall rules so that ipv6 traffic is allowed, most importantly icmp6, details covered in the faq.

So, now I have a tunnel setup, the next step is to find a registrar to move my domains to whom supports IPv6 records, surprisingly the big boys, tucows, 1&1, go daddy don’t (well go daddy allows the creation of AAAA records but you can’t use them for your NS records).
The FAQ once again has this covered here
Once I’ve transfered my domains across, I’ll be ditching OpenBSD 4.3 & moving to -CURRENT temporarily as Apache supports IPv6 there.

RIP Jun-ichiro “itojun” Hagino

Chillispot PF Config & More

I have just submitted a PR to update my port for FreeBSD
The update contains a Install guide which covers install & config of Chillispot, Apache 1.3.x with mod_ssl, MySQL 4.1.x & freeRADIUS + A sample PF config file aswell as a IPFW config file & a RC startup script.
If you can’t wait till it’s merged into the tree you can grab a copy of the port from here

Dovecot on OpenBSD file_lock_dotlock() errors

If after installing Dovecot on OpenBSD you get the following error when you try to access your mailbox:
open(/var/mail/.temp.host.1234.abcdefg) failed: Permission denied
file_lock_dotlock() failed with mbox file /var/mail/user: Permission denied

then uncomment & change the mbox_write_locks entry in /etc/dovecot from mbox_write_locks = dotlock fcntl to mbox_write_locks = fcntl

Everything should spring to life afterwards! 🙂

NetBoot OS X from a OpenBSD Server & NetInstall from a OpenDarwin NFS Server PT2

Get the PDF version of the guide here
All Info in this guide was sourced from the following pages (thnx guys) & the patch is a mod of Mike Passwalls original patch for linux
http://homepage.mac.com/nand/macosx/netboot.html (not english)
http://frank.gwc.org.uk/~ali/nb/
http://www.lysator.liu.se/~/torkel/computer/netboot-macosx.html
http://mike.passwall.com/macnc/

ToDo
Make a patch for dhcpd on OpenBSD 3.6
Make the whole thing run on OpenBSD

NetBoot OS X from a OpenBSD Server & NetInstall from a OpenDarwin NFS Server

This project is still yet to be finished, at the moment Im using 2 box’s to carry out the installation, the aim is to have one box running OpenBSD doing everything (unfortunately there is no HFS support within the o/s & Im having problems getting mountd to accept connections from clients on a non reserved port)
I did look at FreeBSD 5.3 with HFS+ support but Its early days for the project thus the system panicked everytime I attempt to copy to the NFS share from another host.

1x PC running OpenBSD which is running a tfptd & hacked DHCPD
1x PC running OpenDarwin which has a HFS formatted volume containing the OS X install files shared via NFS
1x Mac (G3 iBook in my case)

Mac gets boot info & kernel image from OpenBSD box & boots, then connects to the OpenDarwin box & starts the GUI/Setup.

I have managed to succesfully install OS X 10.3 & 10.4 with this setup though how the install files where shared on the OpenDarwin box varied between the NetInstall of 10.3 & 10.4.

Lets go through the core part of the setup which needs to be done independent of which version of OS X you are going to be installing.

1. Install OpenDarwin, as OpenDarwin x86 runs off a UFS partition you’ll need a 2nd partition (atleast 2.2gigs if youre installing 10.4) which you’ll format as HFS so remember to partition manually. Note the partition number you’ve installed onto as you’ll need it in the next step!

2. Upon 1st boot you’ll have to manually specify the location of the root partition manually as OpenDarwin doesnt seem to find it & sits there idle.
Press enter at the prompt to specify boot time options & at the prompt enter
rd=disk#s# convention being disk “disknumber” s “partition number”
Once youre logged in edit /Library/Preferences/SystemConfiguration/com.apple.Boot.plist & add rd=disk#s# in the string section under the kernel flags key.

3. Now format the 2nd partition using the newfs_hfs tool
newfs_hfs -v pickaname /dev/disk#s#

4. reboot & log back in, if you look in /Volumes/ you should have a folder called pickaname (or whatever name you picked :P)

5. Using the niutil (netinfo util) you need to create a NFS share
niutil . -create /exports/Volumes/pickaname opts maproot=root:wheel
this will create a share accessible by any host to allow specific hosts use the following command:
niutil . -create /exports/Volumes/pickaname clients 192.168.0.bla
to add aditional IP addresses use the append switch:
niutil . -append /exports/Volumes/pickaname clients 192.168.0.bla

6. To start sharing run:
portmap
nfsd -t -u -n 4
mountd

you may want to add these commands to your /etc/rc to save you having to run it everytime.

7. Run ifconfig -a & note the MAC address of your network card.

1.Install OpenBSD 3.5 (in any configuration you like)
2. Download & extract the sources into /usr/src from the the OpenBSD ftp site
3. Download the patch for dhcpd
& apply to source
patch -p0 < obsd_35patch

4. goto /usr/src/usr.sbin/dhcp/server & run make
5. make a backup copy of your original dhcpd & then overwrite with your new copy
cp /usr/sbin/dhpd /usr/sbin/dhcpd.original
cp dhcpd /usr/sbin/

6. With your dhcpd in place, its onto creating the dhcp lease info. open /etc/dhcpd.conf in your editor & paste the following in & edit to your requirements, you’ll need the MAC addresses of your Mac & PC running OpenDarwin

shared-network LOCAL-NET {
option domain-name "domainname.co.uk";
option domain-name-servers 194.168.4.100, 194.168.8.100;
subnet 192.168.0.0 netmask 255.255.255.0
{option routers 192.168.0.1;
range 192.168.0.2 192.168.0.16;
default-lease-time 600;
max-lease-time 7200;
allow bootp;
not authoritative;
}
host ibook {
hardware ethernet 00:03:66:55:cf:b8;
fixed-address 192.168.0.33;
filename "BootX";
server-name "192.168.0.1";
}
host darwin {
hardware ethernet 00:04:55:66:dd:b5;
fixed-address 192.168.0.10;
}
}

7. Edit /etc/dhcpd.interfaces & enter the name of the interface which dhcpd will run on, run
ifconfig -a if youre unsure of which interface.
8. Edit /etc/bootparams & specify the locations of the root & private folders that the mac will mount on boot
the convention is
hostname root=path private=path eg
ibook root=192.168.0.10:/Volumes/pickaname private=192.168.0.10:/Volumes/pickaname

9.Now onto enabling the services on boot, open /etc/rc.conf.local in your editor & add the following lines:

bootparamd_flags=""
dhcpd_flags="-q"

then open /etc/inetd.conf & uncomment
tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot

10. You’ll need to create a folder on the root of your disc called tftpboot, this folder is going to store the files to boot your mac.
11. Using your Mac or the OpenDarwin box copy the following files from your OS X disks to /tftpboot on your openbsd box:
System/Library/CoreServices/BootX
mach_kernel rename it to: mach.macosx
Extensions.mkext rename it to mach.macosx.mkext

To Install OSX 10.3 (Panther)
As the install is spread over multiple discs & the system reboots after the 1st CD is finished, I didnt bother trying to get a full install going at once, Instead I installed the Core & BSD componenets, then rebooted, mounted the NFS share & installed the other components by hand.
1.Copy the contents of CD1 to your nfs share
pax -r -w -p e /Volumes/Mac OS X Install Disc 1/* /Volumes/pickaname/
2. On your mac you’ll need to set the following variables either at the openfirmware prompt directly or using the nvram tool within OS X

boot-device enet:192.168.0.1
boot-args rf=nfs:192.168.0.10:/Volumes/pickaname

If the installer complains that there is 0 space available on your Mac to install onto then make sure you have a folder called .vol on your NFS share.

Theoretically is should be possible to install Tiger this way aswell but the installer complains that the harddisk on the Mac cannot be installed onto as the system cannot be started from that volume!!!

To Install OSX 10.4 (Tiger)
Simply copy the .dmg of the latest Beta Seed to the /Volumes/pickaname
On your mac you’ll need to set the following variables either at the openfirmware prompt directly or using the nvram tool within OS X

boot-device enet:192.168.0.1
boot-args rf=nfs:192.168.0.10:/Volumes/pickaname:nameoftigerimage.dmg

It should be possible to install 10.3 this way aswell though I havent tried.
If youre planing on only installing from a disk image then theoretically there is no need to create a HFS partition on the OpenDarwin box & If you can get OpenBSD to accept connections from clients on non reserved ports then the OpenDarwin box can be ditched all together.

Darwin Streaming Server Port Usage

As Darwin SS uses the RTSP protocol to stream (unless youre streaming via port 80) you must have the following ports open on your firewall 554, 7070, 8000, 8001 then anything starting from 6970 & up depending on how many clients you want to support.
Apple have a basic support article on this here