networking – GeekLAN

07/05/201608/05/2016

Running FreeBSD / OpenBSD / NetBSD as a virtualised guest on Online.net

I’ve been running a mixture of FreeBSD / OpenBSD & NetBSD as guests on a dedicated server at Online.net. While getting the operating systems installed was fairly seamless, getting networking going was not.

Client are not isolated in a layer 2 domain
DHCPv6 config is broken

Clients not being isolated is not so much a problem itself and is typically what you’d expect if you plugged a bunch of computers into a switch with a single VLAN or unmanaged switched for example; but in a shared environment with untrusted tenants it can cause problems. Broadcast & IPv6 multicast floods aside, one is open to most of the attacks in something like THC-IPv6 due to lack of MLD snooping which would prevent a rogue IPv6 router.

Attacks via IPv6 are not so much of a problem as their use of non-RFC complaint timers settings in their DHCPv6 make it unfeasible to use the offered native IPv6 connectivity as clients will fail to renew leases. Depending on the DHCPv6 client used, the amount of time it takes fail to renew a lease will vary. dhcpcd for example now warns if detects a lease is not compliant with RFC 3315 section 22.4 “Identity Association for Non-temporary Addresses Option”.

Despite having a vast address range in IPv6 and a /48 subnet is allotted free of charge, you’ll need the equal amount of v4 address addresses as the v6 addresses you intend to use at Online.net. There is a way of using a /48 and allocating addresses yourself but it’s only possible using a version of Proxmox which they provide.

You can save yourself a lot of hassle both with configuration & trying to deal with their support regarding IPv6 by using a Hurricane Electric tunnel. I actually found connectivity was also faster from Hurricane Electric than using the native connectivity.

For IPv4 connectivity on a guest (assuming you’re renting individual IP addresses & not a /27 prefix), you’ll need to use the default gateway IP address assigned to your host alongside the allotted IP address and a /32 prefix.

Assuming the network details are as follows
Default gateway on host: 192.0.2.1
Failover IP #1: 198.51.100.10, assigned to MAC address 00:50:56:00:01:AA
Failover IP #2: 203.0.113.11, assigned to MAC address 00:50:56:00:02:BB
Failover IP #3: 203.0.113.100, assigned to MAC address 00:50:56:00:03:CC

The MAC addresses need to be assigned to the tap(4) interface on the host.
If you’re using bhyve and your guest is using the interface tap0, this would be performed using the -s flag to configure the virtual PCI ethernet card, eg -s 1:0,virtio-net,tap0,mac=00:50:56:00:01:AA

It’s then onto configuring each OS to handle a gateway which is in a another subnet for IPv4 connectivity.

FreeBSD

In FreeBSD you need to construct a route to reach the default IP address first, before you specify the default IP address, otherwise things will not work. So assuming we’re going to use Failover IP #1, your configuration in /etc/rc.conf would be as follows

ifconfig_vtnet0="inet 198.51.100.10/32" gateway_if="vtnet0" gateway_ip="192.0.2.1" static_routes="gateway default" route_gateway="-host $gateway_ip -interface $gateway_if" route_default="default $gateway_ip"

Note, the installer at present prevents network installs, you should use a iso image containing the distfiles, bug 206355 has more details.

NetBSD

On NetBSD, configure networking using /etc/netstart.local, entering the commands you’d enter at the console inside the file. Assuming failover IP #2 is going to be used for the NetBSD VM, the following would configure the guest to reach the outside world using 192.0.2.1, as discussed in the NetBSD Network FAQ

ifconfig vioif0 203.0.113.11/32 route add -net 192.0.2.1 -link -cloning -iface vioif0 route add default -ifa 203.0.113.11 192.0.2.1

OpenBSD

On OpenBSD, configure the networking from the ethernet interfaces configuration file hostname.if(5).

Assuming failover IP #3 is going to be used for the OpenBSD VM, the following will setup networking.

/etc/hostname.vio0

inet 203.0.113.100 255.255.255.255 NONE !/sbin/route add -net 192.0.2.1 -netmask 255.255.255.255 -link -cloning -iface vio0 !/sbin/route add default -ifa 203.0.113.100 192.0.2.1

It’s also possible to not specify the -cloning flag but a patch is required if you’re running 5.9 release.

02/12/201503/12/2015

Building a l2tp/IPsec VPN based around a OpenBSD head-end – Part 1

This is the first in a series of posts to cover building a l2tp/IPsec VPN service which remote users (road warriors) connect to.
In this post I will begin with getting OpenBSD setup as the head-end & follow up with subsequent posts to cover configuration of various platforms as clients which compose the road warriors.
Undeadly featured an article on configuring OpenBSD in 2012, things have improved since this article was posted and some of the steps are no longer required, hence I will go over the process again here.

It’s assumed you have an install of OpenBSD running that’s setup as a gateway and communicating on the network, we will continue from there.

The following snippet of config needs to be added to your PF config (/etc/pf.conf by default). It unconditionally permits the IPsec ESP & AH protocols intended for the OpenBSD host, as well as any UDP traffic for ISAKMP and to support NAT traversal.
pass quick proto { esp, ah } from any to self pass quick proto udp from any to self port {isakmp, ipsec-nat-t} keep state pass on enc0 from any to self keep state (if-bound)

A minimal PF config which just permits the establishment of a VPN tunnel might look like the following

set skip lo block return pass quick proto { esp, ah } from any to self pass quick proto udp from any to self port {isakmp, ipsec-nat-t} keep state pass on enc0 from any to self keep state (if-bound)

By only permitting isakmp, it enforces having a working IPsec config before anything else happens whereas permitting UDP port 1701 would permit the establishment of a l2tp tunnel without IPsec which in this scenario would likely be undesired.

A basic IPsec config to use a pre-shared key.The default ciphers used for main & quick mode are documented in ipsec.conf(5). The IP address 1.2.3.4 is configured on the OpenBSD host which connections will be accepted on.

ike passive esp transport proto udp from 1.2.3.4 to any port 1701 psk "password"

Note, the OpenBSD defaults are too high for establishing a connection using the networking preferences on Apple devices and so would need to be restricted down to auth "hmac-sha1" enc "3des" group modp1024 which is not recommended, configuring Apple systems will be covered as a separate article.

The default npppd config (/etc/npppd/nppd.conf) works as-is, without any further changes required. That is unless you prefer to use RADIUS for accounting, instead of local user accounts.

myuser:\ :password=mypass:\ :framed-ip-address=10.0.0.111:

npppd is set to use pppx(4) interfaces for established sessions, in order for these interfaces to work correctly, pipex(4) needs to be enabled.

sysctl net.pipex.enable=1

and adding net.pipex.enable=1 to /etc/sysctl.conf so it’s set on boot.

Note, hosts missing this commit (5.8-RELEASE and snapshots from today & prior) will suffer a panic on the OpenBSD host upon establishment of a session by clients, if pipex(4) is not enabled.

Start isakmpd & npppd with

isakmpd -K
npppd

Load your ipsec.conf with
ipsecctl -f /etc/ipsec.conf

Your host should be ready to accept VPN connections, set this services to be started on boot by adding the following to /etc/rc.conf.local
isakmpd_flags="-K" ipsec=YES npppd_flags=""

01/05/201501/05/2015

Captive Portals & Brighton

Yesterday I gave a talk at SANE user group on my history with wireless networks as part of the PierToPier.net project in Brighton (now defunct) and my experimentation with captive portal software which I began revisiting this time last year. I thought it would be a good opportunity to develop my programming skills by tidying up and modernising parts of the codebase which caused problems, such as things preventing builds on a modern system with clang which is now the default compiler for FreeBSD on i386/AMD64 architectures and OS X. The slides from my talk can be found here.

Back in the early to mid 2000’s there were 2 initiatives to provide public access wifi in Brighton. Loose Connection and PierToPier.net, each had a different focus & approach.
Loose Connection was a commercial venture which could be deemed a VAR, they resold a ADSL connection along with a draytek router & that was it. Individual wireless networks with the loose connection SSID dotted around drinking holes in Brighton, the founding(?) company Metranet lives on as a WISP today.
PierToPier.net was a community driven effort with a technical team of volunteers, predominantly from a service provider / telecoms / networking background. Each node on the network was sponsored by a host who’d buy and run the equipment while the project members managed it.

The project started off based around the fanless VIA mini-itx boards, Prism2 chipset wireless cards, booting linux with hostapd and nocatauth/nocatsplash off a CF card in a IDE to CF adapter.
This was a very flexible platform, if there was no package for it you could build it with ease, problem was that it had a high cost for entry, £250 to £300? so from the start we were looking to reduce costs.
The other issue was though we’d eliminated moving parts, the casing was not suitable for outdoor use.

With the availability of 3rd party firmware and promotional sales of the WRT54G, PierToPier switched hardware platforms as the low cost solution for new nodes.
Tom Grifiths discovered Chillispot around the same time frame and we adopted it due to enhanced functionality it provided, such as RADIUS accounting and working captive portal. We’d previously ran into issues with browser support running nocatauth which by that point was no longer maintained and stability issues with nocatsplash.

Glastonbury 2005
VIA motherboard again this time with CF adapter & mini-pci slot onboard
2x Atheros A/B/G mini-pci cards, one on mini-pci to PCI bridges and second onboard
Stuck to a pelican case with epoxy
Two holes drilled in the side of the case for external antennas

It was early times for support of the cards and the wireless standards. In this era OpenBSD was leading the way in terms of support of hardware and development of their ieee80211 wifi stack, they were the first to reverse engineer the Atheros binary blob HAL (years before anyone?) but late to the game for the 802.11g, 11a was enabled from the start but didn’t appear to work – bringing the interface up in .11a hostap mode wouldn’t necessarily work.
Looking at alternatives there was a short lived live environment named WifiBSD which was based around FreeBSD but later moved to NetBSD before development ceased. The support for the Atheros cards was not as good as OpenBSD, hence not wasn’t much use.

The hardware for the Glastonbury nodes were truly terrible, all functionality had been wired to a single bus which caused the system to lock hard in most configurations. e.g. you booted from a CF card and tried to bring up a wireless interface. The only way to use the system was to disable everything that wasn’t needed in the BIOS including VGA, if there was an issue, you’d have to factory reset the BIOS before diagnosing.

For the wireless network at Glastonbury, the 11a 5GHz network was used as the backhaul while the 11b/g interface was used for connecting wireless clients. No captive portal, connect to the AP & off you go.

We arrived onsite on Tuesday, starting getting things running, Thursday morning the rain and lightning started things went downhill from there. loss of connectivity between the backhaul links meant things fell apart.

By the time we’d discovered Chillipot the project had a 1.0 release out which had preliminary support for FreeBSD. The website claimed only FreeBSD 5 and up were supported, I created a port and submitted it for inclusion in the tree, net-mgmt/chillispot was born, Edwin@ from the ports team fixed the code so that it’d work on previous releases. I then moved onto creating a OpenBSD port, this was slightly harder and the final peace was actually resolved by a Steve Davies. I got the code to build on OpenBSD but networking wouldn’t work. This turned out to be because an additional 4 bytes needed to be allocated which Steve fixed. It never made it into the OpenBSD tree (only tested it on i386 and SPARC, it used strcpy() everywhere and didn’t run on SPARC) but it can be found in ports-wip. I then moved onto creating a live CD environment based on FreeBSD 6 using freesbie for advocacy purposes named BrightonChilli. The idea was to remove the hurdle of going through the installation process and provided an environment that just needed the configuration of network interfaces and chillispot. A person with previous experience of running chillispot would be familiar and a new user would not be too out of place.
This was in the days of X configuration being a part of sysinstall which could hang if Xconfigure was run and you’d have to start the install process again as the install was incomplete (for a newcomer). I was interviewed on BSDtalk #73 regarding BrightonChilli.

PierToPier also produced its own Linux image named Muddy Linux targeted for x86 hardware that ran the necessary stack to serve as a node on the network.
After Chillispot 1.1.0, the project went quiet, there was no answer from the founding developer for quite a while and eventually the web hosting stopped and the domain expired.
The community rehomed to coova.org and development continued in Coovachilli which was founded by David Bird, a contributor to Chillispot.
Coovachilli initially lacked support for FreeBSD but it was eventually added in by David and net-mgmt/coovachilli was born in ports. Not much else was done after that until a year ago. With FreeBSD 10 and the switch to clang, the codebase needed attention, first step was to get it to build correctly with GCC. The use of error_t from glibc caused the build to fail as it’s not available in FreeBSD, ensuring this was declared allowed the build to complete successfully. To resolve build issues with clang, nested functions were separated out. Any function with missing prototypes & parameter lists were addressed next.
struct ifreq had been marked as deprecated since 2000 and was finally removed in FreeBSD 10. The *BSD specific sections of Coova were switched out to the new struct ifaliasreq & Linux was left to use the pre-existing method. There was extensive use of macros for the logging functionality, these were dropped in favour of using the existing standard syslog(3) with the correct log level defined. This had the benefit of revealing issues which were not detected previously such as incorrect format specifiers.
There are still many things that need to be cleared up, the 3rd party functions added in are particularly problematic and will probably be my next task to replace with standard components.
CoovaChilli & Chillisport have seen large scale deployments thanks to use by Fon, o2 and Google which now owns Coova.

31/07/201409/03/2015

Using ifstated to monitor links and dynamically adjust PF config on event

It’s possible to misuse NAT to load balance outbound traffic across multiple internet connections from different service providers,see the Load Balance Outgoing Traffic section of PF FAQ.
The shortfall with this configuration is when implemented alongside unstable links, forwarding will continue to be attempted over the links which are down, this will cause issues such as long hangs for users behind the NAT while connections time out. To mitigate this, ifstated can be used to smooth things over.
ifstated can be used to run tests & on event perform tasks, if you’re familiar with Cisco IOS, this is similar to some of what is available in EEM. In this scenario, ifstated will be set to ping each gateway at the service provider end of each link every 10 seconds & upon failure, adapt the configuration so traffic is not forwarded down that link. ifstated will continue to perform the tests & when tests start passing because link has re-established successfully, ifstated will reconfigure the system again so links are utilised.

For this post we’ll use the example ruleset from the PF FAQ and adapt it so it can be manipulated by ifstated.

Original pf.conf
lan_net = "192.168.0.0/24" int_if = "dc0" ext_if1 = "fxp0" ext_if2 = "fxp1" ext_gw1 = "198.51.100.100" ext_gw2 = "203.0.113.200"


# nat outgoing connections on each internet interface

match out on $ext_if1 from $lan_net nat-to ($ext_if1)

match out on $ext_if2 from $lan_net nat-to ($ext_if2)
# default deny

block in

block out
# pass all outgoing packets on internal interface

pass out on $int_if to $lan_net

# pass in quick any packets destined for the gateway itself

pass in quick on $int_if from $lan_net to $int_if

# load balance outgoing traffic from internal network.

pass in on $int_if from $lan_net \

route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } \

round-robin

# keep https traffic on a single connection; some web applications,

# especially "secure" ones, don't allow it to change mid-session

pass in on $int_if proto tcp from $lan_net to port https \

route-to ($ext_if1 $ext_gw1)
# general "pass out" rules for external interfaces

pass out on $ext_if1

pass out on $ext_if2

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for # $ext_if2 and $ext_gw2 pass out on $ext_if1 from $ext_if2 route-to ($ext_if2 $ext_gw2) pass out on $ext_if2 from $ext_if1 route-to ($ext_if1 $ext_gw1)

Modified pf.conf
lan_net = "192.168.0.0/24" int_if = "dc0" ext_if1 = "fxp0" ext_if2 = "fxp1" ext_gw1 = "198.51.100.100" ext_gw2 = "203.0.113.200"


# nat outgoing connections on each internet interface

anchor nat-isp1

anchor nat-isp2
set skip on lo
# default deny

block in

block out
anchor "ftp-proxy/*"
# pass all outgoing packets on internal interface

pass out on $int_if to $lan_net

# pass in quick any packets destined for the gateway itself

pass in quick on $int_if from $lan_net to $int_if

# load balance outgoing traffic from internal network.

anchor loadbalance
# keep https traffic on a single connection; some web applications,

# especially "secure" ones, don't allow it to change mid-session

anchor applications
# general "pass out" rules for external interfaces

pass out on $ext_if1

pass out on $ext_if2

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for # $ext_if2 and $ext_gw2 anchor pass-isp1 anchor pass-isp2

The rules for NAT, load balancing & routing are replaced with anchors, ifstated will use these anchors to add & manipulate rules.

ifstated.conf
isp1 = '( "ping -q -c 1 -w 1 -S 198.51.100.199 198.51.100.100 >/dev/null" every 10)'


#If inteface is configured dynamically via dhcp use this instead

#isp2 = '( "ping -q -c 1 -w 1 -S `ifconfig vr2 inet |awk \'/inet/ { print $2 }\'` `awk \'/routers/ { print $3 }\' /var/db/dhclient.leases.vr2 |tail -1 |sed \'s/;//\'`>/dev/null" every 10)'
isp2 = '( "ping -q -c 1 -w 1 -S 203.0.113.220 203.0.113.200 >/dev/null" every 10)'
state allworking {

init {

run 'pfctl -a loadbalance -F rules'

run 'pfctl -a applications -F rules'

run 'pfctl -a nat-isp1 -F rules'

run 'pfctl -a nat-isp2 -F rules'

run 'pfctl -a pass-isp1 -F rules'

run 'pfctl -a pass-isp2 -F rules'
run 'route change default 203.0.113.200'
run 'echo "pass in on vr1 from 192.168.1.0/24 \

route-to { (vr0 198.51.100.100), (vr2 203.0.113.200) } round-robin" | pfctl -a loadbalance -f -'
run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'
run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'
run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'
run 'echo "pass out on vr0 from vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'
run 'echo "pass out on vr2 from vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'

}

if ! $isp1

set-state noisp1

if ! $isp2

set-state noisp2

}
state noisp1 {

init {

run 'pfctl -a loadbalance -F rules'

run 'pfctl -a applications -F rules'

run 'pfctl -a nat-isp1 -F rules'

run 'pfctl -a nat-isp2 -F rules'

run 'pfctl -a pass-isp2 -F rules'

run 'pfctl -a pass-isp1 -F rules'
run 'route change default 203.0.113.200'
run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr2 203.0.113.200) }" | pfctl -a loadbalance -f -'
run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'
run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'
run 'echo "pass out on vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'

}

if $isp1

set-state allworking

if ! $isp2

set-state alldown

}
state noisp2 {

init {

run 'pfctl -a loadbalance -F rules'

run 'pfctl -a applications -F rules'

run 'pfctl -a nat-isp1 -F rules'

run 'pfctl -a nat-isp2 -F rules'

run 'pfctl -a pass-isp2 -F rules'

run 'pfctl -a pass-isp1 -F rules'
run 'route change default 198.51.100.100'
run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr0 198.51.100.100) }" | pfctl -a loadbalance -f -'
run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr0 198.51.100.100)" | pfctl -a applications -f -'
run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'
run 'echo "pass out on vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'

}

if ! $isp1

set-state alldown

if $isp2

set-state allworking

}

state alldown { init { run 'pfctl -a loadbalance -F rules' run 'pfctl -a applications -F rules' run 'pfctl -a nat-isp1 -F rules' run 'pfctl -a nat-isp2 -F rules' run 'pfctl -a pass-isp2 -F rules' run 'pfctl -a pass-isp1 -F rules' } if $isp1 && ! $isp2 set-state noisp2 if $isp2 && ! $isp1 set-state noisp1 if $isp1 && $isp2 set-state all working }

As ifstated is initialised & when it switches states, it flushes the anchors in the pf.conf, sets the default gateway so the host itself can be reachable remotely on the WAN and then injects rules into the PF anchors.

18/03/201225/12/2012

Juniper SRX & FreeBSD/mips

I didn’t realise the Juniper SRX line (at least the 100) was based on a MIPS SoC made by OCTEON.

CPU in a SRX100b
OCTEON CN5020-SCP pass 1.1, Core clock: 500 MHz, DDR clock: 266MHz (532 Mhz data rate)

dmesg from SRX100

Thinking about it now, I now understand why Juniper contributed the code back up to FreeBSD back in 2007 & as I search around for reference material to link to in this blog post the pieces are falling into place.
An announcement was made at the start of month that DTrace had been ported to FreeBSD/MIPS by Oleksandr Tymoshenko.
What this will mean is that when the code makes it back into a Junos release you will have the ability to get near realtime answers of what is going on your router or firewall for example using the network provider & it’ll be safe to run in production because DTrace is designed not to be harmful, something which Cisco doesn’t do & use of debug commands is discouraged on production systems because they are considered harmful.

If you’ve never played with DTrace & have a Mac, its available on all system running Leopard & above, see this article on getting started
Its available in Solaris (& derivatives) which is also where it originates from & on FreeBSD but system has to be rebuilt to enable support, see the wiki article for details.