Running FreeBSD / OpenBSD / NetBSD as a virtualised guest on Online.net

I’ve been running a mixture of FreeBSD / OpenBSD & NetBSD as guests on a dedicated server at Online.net. While getting the operating systems installed was fairly seamless, getting networking going was not.

  1. Client are not isolated in a layer 2 domain
  2. DHCPv6 config is broken

Clients not being isolated is not so much a problem itself and is typically what you’d expect if you plugged a bunch of computers into a switch with a single VLAN or unmanaged switched for example; but in a shared environment with untrusted tenants it can cause problems. Broadcast & IPv6 multicast floods aside, one is open to most of the attacks in something like THC-IPv6 due to lack of MLD snooping which would prevent a rogue IPv6 router.

Attacks via IPv6 are not so much of a problem as their use of non-RFC complaint timers settings in their DHCPv6 make it unfeasible to use the offered native IPv6 connectivity as clients will fail to renew leases. Depending on the DHCPv6 client used, the amount of time it takes fail to renew a lease will vary. dhcpcd for example now warns if detects a lease is not compliant with RFC 3315 section 22.4 “Identity Association for Non-temporary Addresses Option”.

Despite having a vast address range in IPv6 and a /48 subnet is allotted free of charge, you’ll need the equal amount of v4 address addresses as the v6 addresses you intend to use at Online.net. There is a way of using a /48 and allocating addresses yourself but it’s only possible using a version of Proxmox which they provide.

You can save yourself a lot of hassle both with configuration & trying to deal with their support  regarding IPv6 by using a Hurricane Electric tunnel. I actually found connectivity was also faster from Hurricane Electric than using the native connectivity.

For IPv4 connectivity on a guest (assuming you’re renting individual IP addresses & not a /27 prefix), you’ll need to use the default gateway IP address assigned to your host alongside the allotted IP address and a /32 prefix.

Assuming the network details are as follows
Default gateway on host: 192.0.2.1
Failover IP #1: 198.51.100.10, assigned to MAC address 00:50:56:00:01:AA
Failover IP #2: 203.0.113.11, assigned to MAC address 00:50:56:00:02:BB
Failover IP #3: 203.0.113.100, assigned to MAC address 00:50:56:00:03:CC

The MAC addresses need to be assigned to the tap(4) interface on the host.
If you’re using bhyve and your guest is using the interface tap0, this would be performed using the -s flag to configure the virtual PCI ethernet card, eg -s 1:0,virtio-net,tap0,mac=00:50:56:00:01:AA

It’s then onto configuring each OS to handle a gateway which is in a another subnet for IPv4 connectivity.

FreeBSD

In FreeBSD you need to construct a route to reach the default IP address first, before you specify the default IP address, otherwise things will not work. So assuming we’re going to use Failover IP #1, your configuration in /etc/rc.conf would be as follows

ifconfig_vtnet0="inet 198.51.100.10/32"
gateway_if="vtnet0"
gateway_ip="192.0.2.1"
static_routes="gateway default"
route_gateway="-host $gateway_ip -interface $gateway_if"
route_default="default $gateway_ip"

Note, the installer at present prevents network installs, you should use a iso image containing the distfiles, bug 206355 has more details.

NetBSD

On NetBSD, configure networking using /etc/netstart.local, entering the commands you’d enter at the console inside the file. Assuming failover IP #2 is going to be used for the NetBSD VM, the following would configure the guest to reach the outside world using 192.0.2.1, as discussed in the NetBSD Network FAQ

ifconfig vioif0 203.0.113.11/32
route add -net 192.0.2.1 -link -cloning -iface vioif0
route add default -ifa 203.0.113.11 192.0.2.1

OpenBSD

On OpenBSD, configure the networking from the ethernet interfaces configuration file hostname.if(5).

Assuming failover IP #3 is going to be used for the OpenBSD VM, the following will setup networking.

/etc/hostname.vio0

inet 203.0.113.100 255.255.255.255 NONE
!/sbin/route add -net 192.0.2.1 -netmask 255.255.255.255 -link -cloning -iface vio0
!/sbin/route add default -ifa 203.0.113.100 192.0.2.1

It’s also possible to not specify the -cloning flag but a patch is required if you’re running 5.9 release.

Building a l2tp/IPsec VPN based around a OpenBSD head-end – Part 1

This is the first in a series of posts to cover building a l2tp/IPsec VPN service which remote users (road warriors) connect to.
In this post I will begin with getting OpenBSD setup as the head-end & follow up with subsequent posts to cover configuration of various platforms as clients which compose the road warriors.
Undeadly featured an article on configuring OpenBSD in 2012, things have improved since this article was posted and some of the steps are no longer required, hence I will go over the process again here.

It’s assumed you have an install of OpenBSD running that’s setup as a gateway and communicating on the network, we will continue from there.

The following snippet of config needs to be added to your PF config (/etc/pf.conf by default). It unconditionally permits the IPsec ESP & AH protocols intended for the OpenBSD host, as well as any UDP traffic for ISAKMP and to support NAT traversal.
pass quick proto { esp, ah } from any to self
pass quick proto udp from any to self port {isakmp, ipsec-nat-t} keep state
pass on enc0 from any to self keep state (if-bound)

A minimal PF config which just permits the establishment of a VPN tunnel might look like the following

set skip lo
block return
pass quick proto { esp, ah } from any to self
pass quick proto udp from any to self port {isakmp, ipsec-nat-t} keep state
pass on enc0 from any to self keep state (if-bound)

By only permitting isakmp, it enforces having a working IPsec config before anything else happens whereas permitting UDP port 1701 would permit the establishment of a l2tp tunnel without IPsec which in this scenario would likely be undesired.

A basic IPsec config to use a pre-shared key.The default ciphers used for main & quick mode are documented in ipsec.conf(5). The IP address 1.2.3.4 is configured on the OpenBSD host which connections will be accepted on.

ike passive esp transport proto udp from 1.2.3.4 to any port 1701 psk "password"

Note, the OpenBSD defaults are too high for establishing a connection using the networking preferences on Apple devices and so would need to be restricted down to auth "hmac-sha1" enc "3des" group modp1024 which is not recommended, configuring Apple systems will be covered as a separate article.

The default npppd config (/etc/npppd/nppd.conf) works as-is, without any further changes required. That is unless you prefer to use RADIUS for accounting, instead of local user accounts.

myuser:\
    :password=mypass:\
    :framed-ip-address=10.0.0.111:

npppd is set to use pppx(4) interfaces for established sessions, in order for these interfaces to work correctly, pipex(4) needs to be enabled.

sysctl net.pipex.enable=1

and adding net.pipex.enable=1 to /etc/sysctl.conf so it’s set on boot.

Note, hosts missing this commit (5.8-RELEASE and snapshots from today & prior) will suffer a panic on the OpenBSD host upon establishment of a session by clients, if pipex(4) is not enabled.

Start isakmpd & npppd with

isakmpd -K
npppd

Load your ipsec.conf with
ipsecctl -f /etc/ipsec.conf

Your host should be ready to accept VPN connections, set this services to be started on boot by adding the following to /etc/rc.conf.local
isakmpd_flags="-K"
ipsec=YES
npppd_flags=""

Using ifstated to monitor links and dynamically adjust PF config on event

It’s possible to misuse NAT to load balance outbound traffic across multiple internet connections from different service providers,see the Load Balance Outgoing Traffic section of PF FAQ.
The shortfall with this configuration is when implemented alongside unstable links, forwarding will continue to be attempted over the links which are down, this will cause issues such as long hangs for users behind the NAT while connections time out. To mitigate this, ifstated can be used to smooth things over.
ifstated can be used to run tests & on event perform tasks, if you’re familiar with Cisco IOS, this is similar to some of what is available in EEM. In this scenario, ifstated will be set to ping each gateway at the service provider end of each link every 10 seconds & upon failure, adapt the configuration so traffic is not forwarded down that link. ifstated will continue to perform the tests & when tests start passing because link has re-established successfully, ifstated will reconfigure the system again so links are utilised.

For this post we’ll use the example ruleset from the PF FAQ and adapt it so it can be manipulated by ifstated.

Original pf.conf

lan_net = "192.168.0.0/24"
int_if = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "198.51.100.100"
ext_gw2 = "203.0.113.200"

# nat outgoing connections on each internet interface
match out on $ext_if1 from $lan_net nat-to ($ext_if1)
match out on $ext_if2 from $lan_net nat-to ($ext_if2)

# default deny
block in
block out

# pass all outgoing packets on internal interface
pass out on $int_if to $lan_net
# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
# load balance outgoing traffic from internal network.
pass in on $int_if from $lan_net \
route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } \
round-robin
# keep https traffic on a single connection; some web applications,
# especially "secure" ones, don't allow it to change mid-session
pass in on $int_if proto tcp from $lan_net to port https \
route-to ($ext_if1 $ext_gw1)

# general "pass out" rules for external interfaces
pass out on $ext_if1
pass out on $ext_if2

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
pass out on $ext_if1 from $ext_if2 route-to ($ext_if2 $ext_gw2)
pass out on $ext_if2 from $ext_if1 route-to ($ext_if1 $ext_gw1)

Modified pf.conf

lan_net = "192.168.0.0/24"
int_if = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "198.51.100.100"
ext_gw2 = "203.0.113.200"

# nat outgoing connections on each internet interface
anchor nat-isp1
anchor nat-isp2

set skip on lo

# default deny
block in
block out

anchor "ftp-proxy/*"

# pass all outgoing packets on internal interface
pass out on $int_if to $lan_net
# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
# load balance outgoing traffic from internal network.
anchor loadbalance

# keep https traffic on a single connection; some web applications,
# especially "secure" ones, don't allow it to change mid-session
anchor applications

# general "pass out" rules for external interfaces
pass out on $ext_if1
pass out on $ext_if2

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
anchor pass-isp1
anchor pass-isp2

The rules for NAT, load balancing & routing are replaced with anchors, ifstated will use these anchors to add & manipulate rules.

ifstated.conf

isp1 = '( "ping -q -c 1 -w 1 -S 198.51.100.199 198.51.100.100 >/dev/null" every 10)'

#If inteface is configured dynamically via dhcp use this instead
#isp2 = '( "ping -q -c 1 -w 1 -S `ifconfig vr2 inet |awk \'/inet/ { print $2 }\'` `awk \'/routers/ { print $3 }\' /var/db/dhclient.leases.vr2 |tail -1 |sed \'s/;//\'`>/dev/null" every 10)'

isp2 = '( "ping -q -c 1 -w 1 -S 203.0.113.220 203.0.113.200 >/dev/null" every 10)'

state allworking {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
run 'pfctl -a pass-isp2 -F rules'

run 'route change default 203.0.113.200'

run 'echo "pass in on vr1 from 192.168.1.0/24 \
route-to { (vr0 198.51.100.100), (vr2 203.0.113.200) } round-robin" | pfctl -a loadbalance -f -'

run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'

run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'

run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'

run 'echo "pass out on vr0 from vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'

run 'echo "pass out on vr2 from vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'
}
if ! $isp1
set-state noisp1
if ! $isp2
set-state noisp2
}

state noisp1 {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'

run 'route change default 203.0.113.200'

run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr2 203.0.113.200) }" | pfctl -a loadbalance -f -'

run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'

run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'

run 'echo "pass out on vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'
}
if $isp1
set-state allworking
if ! $isp2
set-state alldown
}

state noisp2 {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'

run 'route change default 198.51.100.100'

run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr0 198.51.100.100) }" | pfctl -a loadbalance -f -'

run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr0 198.51.100.100)" | pfctl -a applications -f -'

run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'

run 'echo "pass out on vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'
}
if ! $isp1
set-state alldown
if $isp2
set-state allworking
}

state alldown {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
}
if $isp1 && ! $isp2
set-state noisp2
if $isp2 && ! $isp1
set-state noisp1
if $isp1 && $isp2
set-state all working
}

As ifstated is initialised & when it switches states, it flushes the anchors in the pf.conf, sets the default gateway so the host itself can be reachable remotely on the WAN and then injects rules into the PF anchors.

OpenBSD on 11″ MacBook Air 5,1 (mid-2012)

While my MacBook was away at the service centre to have the SSD replaced I noticed the NetBSD wiki had marked the install guide as obsolete as it’s no longer required to build a custom kernel because the necessary changes have been integrated so that the generic kernel works out of the box.
The last time I tried to run OpenBSD on a MacBook Air was over the christmas holiday on a mid-2012 13″ model & while I managed to boot a multiuser system, USB support was very unstable & eDP support was missing from Xenocara.
Having received my MacBook back I decided to revisit Net/OpenBSD, I tried booting NetBSD/AMD64 6.1.1 & the 29/08/2013 AMD64 OpenBSD snapshot.
Both exhibited the same behavior, as soon as the kernel loaded into memory the screen would go blank, I attached a thunderbolt display which made displayed some output before going to a blank screen as the kernel probes for devices, I believe this is when OpenBSD now changes font.
I switched from the OpenBSD snapshot to what I (half asleep) thought was 5.4-RELEASE but it turned out to be a mislabelled iso of a snapshot from July. This time it worked fine using a Thunderbolt display, screen still goes blank otherwise.
Onboard wireless doesn’t work, instead I’m using a tiny urtwn(4) wireless adapter.

20130905-021145.jpg

OpenBSD 5.4-current (GENERIC.MP) #50: Mon Sep 2 13:43:54 MDT 2013
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
RTC BIOS diagnostic error b1
real mem = 8475713536 (8083MB)
avail mem = 8242003968 (7860MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0000 (54 entries)
bios0: vendor Apple Inc. version "MBA51.88Z.00EF.B02.1211271028" date 11/27/2012
bios0: Apple Inc. MacBookAir5,1
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP HPET APIC SBST ECDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT DMAR MCFG
acpi0: wakeup devices P0P2(S4) PEG2(S4) EC__(S4) HDEF(S4) RP02(S4) ARPT(S4) RP05(S4) EHC1(S4) EHC2(S4) XHC1(S4) ADP1(S4) LID0(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz, 1896.01 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: apic clock running at 99MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz, 1895.70 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz, 1895.70 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 1, core 0, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz, 1895.70 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpiec0 at acpi0
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-153
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (P0P2)
acpiprt2 at acpi0: bus -1 (PEG2)
acpiprt3 at acpi0: bus 2 (RP02)
acpiprt4 at acpi0: bus 3 (RP05)
acpicpu0 at acpi0: C3, C1, PSS
acpicpu1 at acpi0: C3, C1, PSS
acpicpu2 at acpi0: C3, C1, PSS
acpicpu3 at acpi0: C3, C1, PSS
acpibat0 at acpi0: BAT0 model "3545797981023400290" type 3545797981528607052 oem "3545797981528673619"
acpiac0 at acpi0: AC unit offline
acpibtn0 at acpi0: LID0
acpibtn1 at acpi0: PWRB
acpibtn2 at acpi0: SLPB
acpivideo0 at acpi0: IGPU
acpivout0 at acpivideo0: DD02
cpu0: Enhanced SpeedStep 1896 MHz: speeds: 2001, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz
memory map conflict 0xe00f8000/0x1000
memory map conflict 0xfed1c000/0x4000
memory map conflict 0xffe70000/0x30000
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 3G Host" rev 0x09
vga1 at pci0 dev 2 function 0 "Intel HD Graphics 4000" rev 0x09
intagp0 at vga1
agp0 at intagp0: aperture at 0x90000000, size 0x10000000
inteldrm0 at vga1
drm0 at inteldrm0
inteldrm0: 1366x768
wsdisplay0 at vga1 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
"Intel 7 Series xHCI" rev 0x04 at pci0 dev 20 function 0 not configured
"Intel 7 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
ehci0 at pci0 dev 26 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 7 Series HD Audio" rev 0x04: msi
azalia0: codecs: Cirrus Logic CS4206, Intel/0x2806, using Cirrus Logic CS4206
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 7 Series PCIE" rev 0xc4: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 1 "Intel 7 Series PCIE" rev 0xc4: msi
pci2 at ppb1 bus 2
"Broadcom BCM43224" rev 0x01 at pci2 dev 0 function 0 not configured
ppb2 at pci0 dev 28 function 4 "Intel 7 Series PCIE" rev 0xc4: msi
pci3 at ppb2 bus 3
ppb3 at pci3 dev 0 function 0 vendor "Intel", unknown product 0x1547 rev 0x03
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 vendor "Intel", unknown product 0x1547 rev 0x03: msi
pci5 at ppb4 bus 5
vendor "Intel", unknown product 0x1547 (class system subclass miscellaneous, rev 0x03) at pci5 dev 0 function 0 not configured
ppb5 at pci4 dev 3 function 0 vendor "Intel", unknown product 0x1547 rev 0x03: msi
pci6 at ppb5 bus 6
ppb6 at pci4 dev 4 function 0 vendor "Intel", unknown product 0x1547 rev 0x03: msi
pci7 at ppb6 bus 55
ppb7 at pci4 dev 5 function 0 vendor "Intel", unknown product 0x1547 rev 0x03: msi
pci8 at ppb7 bus 104
ppb8 at pci4 dev 6 function 0 vendor "Intel", unknown product 0x1547 rev 0x03: msi
pci9 at ppb8 bus 105
ehci1 at pci0 dev 29 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 22
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 vendor "Intel", unknown product 0x1e56 rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 7 Series AHCI" rev 0x04: msi, AHCI 1.3
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: SCSI3 0/direct fixed naa.0000000000000000
sd0: 115712MB, 512 bytes/sector, 236978176 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 7 Series SMBus" rev 0x04: apic 2 int 18
iic0 at ichiic0
iic0: addr 0x2c 03=fc 05=66 06=40 71=06 72=80 86=70 90=37 91=1c 92=35 93=3f 94=62 95=8c 96=63 97=85 98=24 99=04 9a=88 9f=7c a0=7f a1=b5 a2=bf a3=7b a4=28 a5=cf a6=64 a7=2d words 00=0000 01=0000 02=00fc 03=fc00 04=0066 05=6640 06=4000 07=0000
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
nvram: invalid checksum
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
uhub2 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uvideo0 at uhub2 port 1 configuration 1 interface 0 "Apple Inc. FaceTime HD Camera (Built-in)" rev 2.00/80.25 addr 3
video0 at uvideo0
ugen0 at uhub2 port 1 configuration 1 "Apple Inc. FaceTime HD Camera (Built-in)" rev 2.00/80.25 addr 3
uhub3 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub4 at uhub3 port 8 "Standard Microsystems product 0x2512" rev 2.00/b.b3 addr 3
uhub5 at uhub4 port 1 "Apple Inc. BRCM20702 Hub" rev 2.00/1.00 addr 4
uhub5: device problem, disabling port 1
uhidev0 at uhub5 port 2 configuration 1 interface 0 "Apple Computer product 0x820b" rev 2.00/1.00 addr 5
uhidev0: iclass 3/1, 2 report ids
ums0 at uhidev0 reportid 2: 3 buttons
wsmouse0 at ums0 mux 0
ugen1 at uhub5 port 3 "Apple Inc. Bluetooth USB Host Controller" rev 2.00/1.00 addr 6
uhidev1 at uhub4 port 2 configuration 1 interface 0 "Apple Inc. Apple Internal Keyboard / Trackpad" rev 2.00/2.19 addr 7
uhidev1: iclass 3/1, 9 report ids
ukbd0 at uhidev1 reportid 1: 8 variable keys, 6 key codes, country code 15
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhid0 at uhidev1 reportid 9: input=0, output=0, feature=3
uhidev2 at uhub4 port 2 configuration 1 interface 1 "Apple Inc. Apple Internal Keyboard / Trackpad" rev 2.00/2.19 addr 7
uhidev2: iclass 3/0, 68 report ids
uhid1 at uhidev2 reportid 68: input=511, output=0, feature=0
uhidev3 at uhub4 port 2 configuration 1 interface 2 "Apple Inc. Apple Internal Keyboard / Trackpad" rev 2.00/2.19 addr 7
uhidev3: iclass 3/1, 2 report ids
ums1 at uhidev3 reportid 2: 3 buttons
wsmouse1 at ums1 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (0f625884de9eca57.a) swap on sd0b dump on sd0b
clock: unknown CMOS layout
hw.sensors.cpu0.temp0=36.00 degC
hw.sensors.cpu1.temp0=36.00 degC
hw.sensors.cpu2.temp0=36.00 degC
hw.sensors.cpu3.temp0=36.00 degC
hw.sensors.acpibat0.volt0=7.50 VDC (voltage)
hw.sensors.acpibat0.volt1=7.23 VDC (current voltage)
hw.sensors.acpibat0.power0=12.26 W (rate)
hw.sensors.acpibat0.watthour0=42.41 Wh (last full capacity)
hw.sensors.acpibat0.watthour1=0.25 Wh (warning capacity)
hw.sensors.acpibat0.watthour2=0.10 Wh (low capacity)
hw.sensors.acpibat0.watthour3=6.90 Wh (remaining capacity), OK
hw.sensors.acpibat0.raw0=1 (battery discharging), OK
hw.sensors.acpiac0.indicator0=Off (power supply)
hw.sensors.acpibtn0.indicator0=On (lid open)

Beginning LaTeX – Typesetting the OpenBSD FAQ

I attended a one day training course held by the UK TUG back in July of last year which introduced beginners to LaTeX.
It was relatively simple to get up & running & we were able to put together basic documents with ease after a little practice.
Slides from the course
Handout from the course
To apply what I’d learnt on the course I decided to typeset the OpenBSD FAQ to get me on my way with LaTeX as the official PDF available for download appears to be generated using a pdf printer from the website which is great (links & chapters are there & working) but I don’t think it look that great.
A beautiful OS deserves beautiful documentation! 🙂
So I had a brief attempt at it the days proceeding the course & got side tracked after doing the very basics on the first 4 chapters. Nearly a year on, I thought I’d have another stab at it.
The tex files are in a Mercurial repo & there’s a PDF too 🙂
I’ve managed to get 10 of the 15 sections from the faq into tex files so far, with basic formatting applied to text, but there are lots to do yet e.g links, tidying up formatting, setting a typographical convention & applying it consistently.

Dell PowerEdge T105 & *BSD

Dell where running a special offer this week on the PowerEdge T105 servers.
For £173inc Vat & Shipping they make perfect test boxes, I placed the order on monday & they where here on thursday.
I’ve spent some of today trying ou the AMD64 flavours of FreeBSD 6.3 & 7.0-RC1, NetBSD 4.0 & 200802010002Z snapshot, OpenBSD 4.2 RELEASE & CURRENT.
One word of warning the onboard broadcom network card is a POS, you will need an additional network card installed in the system if you’re planning to have any means of connectivity to you box.
I used a cheapo intel pro/1000 GT PCI network card.

Here are some dmesgs:
FreeBSD 6.3-RELEASE AMD64
FreeBSD 7.0-RC1 AMD64
The broadcom network card was enabled in the bios but wasn’t detected by the kernel

I was unable to NetBSD 4.0 & 200802010002Z as the setup program claimed there where any disks installed.

OpenBSD 4.2-RELEASE GENERIC kernel
OpenBSD 4.2-RELEASE GENERIC.MP kernel
OpenBSD 4.2-CURRENT GENERIC kernel
The broadcom network worked fine during the install process as far as I was able to obtain a IP address from a DHCP server, upon reboot when the system went multiuser & the network card was initialised the system would panic, using the intel card instead stopped the panic onboot, but still panicked on reboot, disabling the broadcom network card in the bios solved any panics. Screenshot
I was unable to test the 4.2-CURRENT GENERIC.MP kernel as the system failed to boot, complaining about em0: watchdog timeout -- resetting
&
wd0a: device timeout writing fsbn 1885728 of 1885728-1885759 (wd0 bn 1885791; cn 11 tn 98 sn 12), retrying Screenshot

I also booted the system off the FreeBSD-CURRENT snapshot using the bootonly iso, the broadcom network card was detected but panicked when attempting to obtain a IP address via DHCP.

Jetway J7F2WE1G5D-OC-PB

6 Months ago I bought a mini itx motherboard to replace my current ancient web server / firewall, I went for the Jetway J7F2WE1G5D-OC-PB as it was cheaper then the VIA ones & it also supports expansion via daughterboards, there’s a whole range to choose from, I went for the AD3RTLAN-G which gives you three additional gigabit interfaces which are based on the Realtek 8169 chipset. Sadly this chipset does have some limitations as mentioned in re(4) on OpenBSD
The RealTek 8169, 8169S and 8110S chips are only capable of transmitting
Jumbo frames up to 7440 bytes in size.

But I’m sure that should be good enough for a network of 1 user! =)

Hopefully within the next couple of weeks I well get OpenBSD 4.2 installed on this box & replace the current server, the only holdup for me atm is the builtin VIA Rhine-II interface doesn’t support adjustment of the mtu, which is going to cause some problems as I’m using pppoe(4) & don’t want use mssfixup in PF, using one of the gigabit interfaces instead would be a waste.

Dmesg from the 21/11/07 snapshot of -CURRENT

Using the CPAN shell / Installing Bundle::CPAN on OpenBSD

Before you can use the CPAN shell on OpenBSD you need to install p5-LWP-UserAgent-Determined from the ports tree/packages.

Otherwise you wont be able to fetch any components properly
eg:

Fetching with Net::FTP:
ftp//cpan.sunsite.ualberta.ca/pub/CPAN/authors/01mailrc.txt.gz
Couldn't fetch 01mailrc.txt.gz from cpan.sunsite.ualberta.ca
Trying with "/usr/bin/lynx -source" to get ftp://CPAN.mirror.rafal.ca/pub/CPAN/authors/01mailrc.txt.gz
gzip: /root/.cpan/sources/authors/01mailrc.txt: unknown suffix: ignored

& the process will bomb out with MD5 checksum errors eg:


Trying with "/usr/bin/lynx -source" to get
ftp://CPAN.mirror.rafal.ca/pub/CPAN/authors/id/A/AN/ANDK/Bundle-CPAN-1.853.tar.gz
gzip: /root/.cpan/sources/authors/id/A/AN/ANDK/Bundle-CPAN-1.853.tar: unknown suffix: ignored
CPAN: Digest::MD5 loaded ok

Trying with "/usr/bin/lynx -source" to get
ftp://CPAN.mirror.rafal.ca/pub/CPAN/authors/id/A/AN/ANDK/CHECKSUMS

Checksum mismatch for distribution file. Please investigate.

Distribution id = A/AN/ANDK/Bundle-CPAN-1.853.tar.gz
CPAN_USERID ANDK (Andreas J. Koenig <andreas .koenig@anima.de%gt;)
CONTAINSMODS
MD5_STATUS
localfile /root/.cpan/sources/authors/id/A/AN/ANDK/Bundle-CPAN-1.853.tar.gz

I'd recommend removing
/root/.cpan/sources/authors/id/A/AN/ANDK/Bundle-CPAN-1.853.tar.gz. Its MD5
checksum is incorrect. Maybe you have configured your 'urllist' with
a bad URL. Please check this array with 'o conf urllist', and retry.

Connecting to a Windows PPTP based VPN through a OpenBSD / PF firewall

To be able to connect to a Windows based PPTP VPN through a OpenBSD firewall you’ll need to make a couple of changes to allow GRE traffic through.
first add the following to /etc/sysctl.conf:
net.inet.gre.allow=1
net.inet.gre.wccp=1
net.inet.mobileip.allow=1

then add the following to the filter section in your /etc/pf.conf:
pass in on $ext_if proto gre all keep state
pass out on $ext_if proto gre all keep state

To make the changes effective without having to reboot issue the following as root:

sysctl net.inet.gre.allow=1
sysctl net.inet.gre.wccp=1
sysctl net.inet.mobileip.allow=1
pfctl -f /etc/pf.conf

Spamd Statistics

After a quick google round I came across this post on misc@
Sadly the link is now dead, but a copy of the script was reposted onto misc@ again which is handy, I’ve also made a copy of the script available here

Anyway, so I copied the script onto one of my openbsd boxes & fired it up resulting in this rather impressive output:
Spamd statistics: (logfile: /var/log/spamd)
Average
Host Seconds Connections (secs/conn)

great!, spamdb lists a huge list of IP addresses & this is all I’m able to get out of it??
After checking out /etc/syslog.conf I found that I hadn’t added the entry for logging spamd when I reformatted, a quick edit & a kill -HUP later things looked much better! 🙂

Spamd statistics: (logfile: /var/log/spamd)
Average
Host Seconds Connections (secs/conn)
201.27.29.243: 12 1 12.00
217.22.88.123: 24 1 24.00

Dovecot on OpenBSD file_lock_dotlock() errors

If after installing Dovecot on OpenBSD you get the following error when you try to access your mailbox:
open(/var/mail/.temp.host.1234.abcdefg) failed: Permission denied
file_lock_dotlock() failed with mbox file /var/mail/user: Permission denied

then uncomment & change the mbox_write_locks entry in /etc/dovecot from mbox_write_locks = dotlock fcntl to mbox_write_locks = fcntl

Everything should spring to life afterwards! 🙂

Switching between XFree86 & Xorg

To switch between the XFree86 X server & the Xorg X Server on OpenBSD simply delete the X symbolic link
rm /usr/X11R6/bin/X
& create a new symbolic link from your preferred X server to X
e.g for Xorg
ln -s /usr/X11R6/bin/Xorg /usr/X11R6/bin/X
or XFree86 SVGA
ln -s /usr/X11R6/bin/XF86_SVGA /usr/X11R6/bin/X

PF Statistics

I’ve gone stat & monitoring crazy in the past few weeks, using Hatchet you can generate graphs & charts from the PF log files on the state of PF. Another tool is also available called pfrtg which does a similar job.

NetBoot OS X from a OpenBSD Server & NetInstall from a OpenDarwin NFS Server PT2

Get the PDF version of the guide here
All Info in this guide was sourced from the following pages (thnx guys) & the patch is a mod of Mike Passwalls original patch for linux
http://homepage.mac.com/nand/macosx/netboot.html (not english)
http://frank.gwc.org.uk/~ali/nb/
http://www.lysator.liu.se/~/torkel/computer/netboot-macosx.html
http://mike.passwall.com/macnc/

ToDo
Make a patch for dhcpd on OpenBSD 3.6
Make the whole thing run on OpenBSD

NetBoot OS X from a OpenBSD Server & NetInstall from a OpenDarwin NFS Server

This project is still yet to be finished, at the moment Im using 2 box’s to carry out the installation, the aim is to have one box running OpenBSD doing everything (unfortunately there is no HFS support within the o/s & Im having problems getting mountd to accept connections from clients on a non reserved port)
I did look at FreeBSD 5.3 with HFS+ support but Its early days for the project thus the system panicked everytime I attempt to copy to the NFS share from another host.

1x PC running OpenBSD which is running a tfptd & hacked DHCPD
1x PC running OpenDarwin which has a HFS formatted volume containing the OS X install files shared via NFS
1x Mac (G3 iBook in my case)

Mac gets boot info & kernel image from OpenBSD box & boots, then connects to the OpenDarwin box & starts the GUI/Setup.

I have managed to succesfully install OS X 10.3 & 10.4 with this setup though how the install files where shared on the OpenDarwin box varied between the NetInstall of 10.3 & 10.4.

Lets go through the core part of the setup which needs to be done independent of which version of OS X you are going to be installing.

1. Install OpenDarwin, as OpenDarwin x86 runs off a UFS partition you’ll need a 2nd partition (atleast 2.2gigs if youre installing 10.4) which you’ll format as HFS so remember to partition manually. Note the partition number you’ve installed onto as you’ll need it in the next step!

2. Upon 1st boot you’ll have to manually specify the location of the root partition manually as OpenDarwin doesnt seem to find it & sits there idle.
Press enter at the prompt to specify boot time options & at the prompt enter
rd=disk#s# convention being disk “disknumber” s “partition number”
Once youre logged in edit /Library/Preferences/SystemConfiguration/com.apple.Boot.plist & add rd=disk#s# in the string section under the kernel flags key.

3. Now format the 2nd partition using the newfs_hfs tool
newfs_hfs -v pickaname /dev/disk#s#

4. reboot & log back in, if you look in /Volumes/ you should have a folder called pickaname (or whatever name you picked :P)

5. Using the niutil (netinfo util) you need to create a NFS share
niutil . -create /exports/Volumes/pickaname opts maproot=root:wheel
this will create a share accessible by any host to allow specific hosts use the following command:
niutil . -create /exports/Volumes/pickaname clients 192.168.0.bla
to add aditional IP addresses use the append switch:
niutil . -append /exports/Volumes/pickaname clients 192.168.0.bla

6. To start sharing run:
portmap
nfsd -t -u -n 4
mountd

you may want to add these commands to your /etc/rc to save you having to run it everytime.

7. Run ifconfig -a & note the MAC address of your network card.

1.Install OpenBSD 3.5 (in any configuration you like)
2. Download & extract the sources into /usr/src from the the OpenBSD ftp site
3. Download the patch for dhcpd
& apply to source
patch -p0 < obsd_35patch

4. goto /usr/src/usr.sbin/dhcp/server & run make
5. make a backup copy of your original dhcpd & then overwrite with your new copy
cp /usr/sbin/dhpd /usr/sbin/dhcpd.original
cp dhcpd /usr/sbin/

6. With your dhcpd in place, its onto creating the dhcp lease info. open /etc/dhcpd.conf in your editor & paste the following in & edit to your requirements, you’ll need the MAC addresses of your Mac & PC running OpenDarwin

shared-network LOCAL-NET {
option domain-name "domainname.co.uk";
option domain-name-servers 194.168.4.100, 194.168.8.100;
subnet 192.168.0.0 netmask 255.255.255.0
{option routers 192.168.0.1;
range 192.168.0.2 192.168.0.16;
default-lease-time 600;
max-lease-time 7200;
allow bootp;
not authoritative;
}
host ibook {
hardware ethernet 00:03:66:55:cf:b8;
fixed-address 192.168.0.33;
filename "BootX";
server-name "192.168.0.1";
}
host darwin {
hardware ethernet 00:04:55:66:dd:b5;
fixed-address 192.168.0.10;
}
}

7. Edit /etc/dhcpd.interfaces & enter the name of the interface which dhcpd will run on, run
ifconfig -a if youre unsure of which interface.
8. Edit /etc/bootparams & specify the locations of the root & private folders that the mac will mount on boot
the convention is
hostname root=path private=path eg
ibook root=192.168.0.10:/Volumes/pickaname private=192.168.0.10:/Volumes/pickaname

9.Now onto enabling the services on boot, open /etc/rc.conf.local in your editor & add the following lines:

bootparamd_flags=""
dhcpd_flags="-q"

then open /etc/inetd.conf & uncomment
tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot

10. You’ll need to create a folder on the root of your disc called tftpboot, this folder is going to store the files to boot your mac.
11. Using your Mac or the OpenDarwin box copy the following files from your OS X disks to /tftpboot on your openbsd box:
System/Library/CoreServices/BootX
mach_kernel rename it to: mach.macosx
Extensions.mkext rename it to mach.macosx.mkext

To Install OSX 10.3 (Panther)
As the install is spread over multiple discs & the system reboots after the 1st CD is finished, I didnt bother trying to get a full install going at once, Instead I installed the Core & BSD componenets, then rebooted, mounted the NFS share & installed the other components by hand.
1.Copy the contents of CD1 to your nfs share
pax -r -w -p e /Volumes/Mac OS X Install Disc 1/* /Volumes/pickaname/
2. On your mac you’ll need to set the following variables either at the openfirmware prompt directly or using the nvram tool within OS X

boot-device enet:192.168.0.1
boot-args rf=nfs:192.168.0.10:/Volumes/pickaname

If the installer complains that there is 0 space available on your Mac to install onto then make sure you have a folder called .vol on your NFS share.

Theoretically is should be possible to install Tiger this way aswell but the installer complains that the harddisk on the Mac cannot be installed onto as the system cannot be started from that volume!!!

To Install OSX 10.4 (Tiger)
Simply copy the .dmg of the latest Beta Seed to the /Volumes/pickaname
On your mac you’ll need to set the following variables either at the openfirmware prompt directly or using the nvram tool within OS X

boot-device enet:192.168.0.1
boot-args rf=nfs:192.168.0.10:/Volumes/pickaname:nameoftigerimage.dmg

It should be possible to install 10.3 this way aswell though I havent tried.
If youre planing on only installing from a disk image then theoretically there is no need to create a HFS partition on the OpenDarwin box & If you can get OpenBSD to accept connections from clients on non reserved ports then the OpenDarwin box can be ditched all together.