GeekLAN

31/07/2014

Using ifstated to monitor links and dynamically adjust PF config on event

Filed under: OpenBSD — Tags: , , — Venture37 @ 11:30 am

It’s possible to misuse NAT to load balance outbound traffic across multiple internet connections from different service providers,see the Load Balance Outgoing Traffic section of PF FAQ.
The shortfall with this configuration is when implemented alongside unstable links, forwarding will continue to be attempted over the links which are down, this will cause issues such as long hangs for users behind the NAT while connections time out. To mitigate this, ifstated can be used to smooth things over.
ifstated can be used to run tests & on event perform tasks, if you’re familiar with Cisco IOS, this is similar to some of what is available in EEM. In this scenario, ifstated will be set to ping each gateway at the service provider end of each link every 10 seconds & upon failure, adapt the configuration so traffic is not forwarded down that link. ifstated will continue to perform the tests & when tests start passing because link has re-established successfully, ifstated will reconfigure the system again so links are utilised.

For this post we’ll use the example ruleset from the PF FAQ and adapt it so it can be manipulated by ifstated.

Original pf.conf

lan_net = "192.168.0.0/24"
int_if = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "198.51.100.100"
ext_gw2 = "203.0.113.200"

# nat outgoing connections on each internet interface
match out on $ext_if1 from $lan_net nat-to ($ext_if1)
match out on $ext_if2 from $lan_net nat-to ($ext_if2)

# default deny
block in
block out

# pass all outgoing packets on internal interface
pass out on $int_if to $lan_net
# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
# load balance outgoing traffic from internal network.
pass in on $int_if from $lan_net \
route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } \
round-robin
# keep https traffic on a single connection; some web applications,
# especially "secure" ones, don't allow it to change mid-session
pass in on $int_if proto tcp from $lan_net to port https \
route-to ($ext_if1 $ext_gw1)

# general "pass out" rules for external interfaces
pass out on $ext_if1
pass out on $ext_if2

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
pass out on $ext_if1 from $ext_if2 route-to ($ext_if2 $ext_gw2)
pass out on $ext_if2 from $ext_if1 route-to ($ext_if1 $ext_gw1)

Modified pf.conf

lan_net = "192.168.0.0/24"
int_if = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "198.51.100.100"
ext_gw2 = "203.0.113.200"

# nat outgoing connections on each internet interface
anchor nat-isp1
anchor nat-isp2

set skip on lo

# default deny
block in
block out

anchor "ftp-proxy/*"

# pass all outgoing packets on internal interface
pass out on $int_if to $lan_net
# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
# load balance outgoing traffic from internal network.
anchor loadbalance

# keep https traffic on a single connection; some web applications,
# especially "secure" ones, don't allow it to change mid-session
anchor applications

# general "pass out" rules for external interfaces
pass out on $ext_if1
pass out on $ext_if2

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
anchor pass-isp1
anchor pass-isp2

The rules for NAT, load balancing & routing are replaced with anchors, ifstated will use these anchors to add & manipulate rules.

ifstated.conf

isp1 = '( "ping -q -c 1 -w 1 -S 198.51.100.199 198.51.100.100 >/dev/null" every 10)'

#If inteface is configured dynamically via dhcp use this instead
#isp2 = '( "ping -q -c 1 -w 1 -S `ifconfig vr2 inet |awk \'/inet/ { print $2 }\'` `awk \'/routers/ { print $3 }\' /var/db/dhclient.leases.vr2 |tail -1 |sed \'s/;//\'`>/dev/null" every 10)'

isp2 = '( "ping -q -c 1 -w 1 -S 203.0.113.220 203.0.113.200 >/dev/null" every 10)'

state allworking {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
run 'pfctl -a pass-isp2 -F rules'

run 'route change default 203.0.113.200'

run 'echo "pass in on vr1 from 192.168.1.0/24 \
route-to { (vr0 198.51.100.100), (vr2 203.0.113.200) } round-robin" | pfctl -a loadbalance -f -'

run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'

run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'

run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'

run 'echo "pass out on vr0 from vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'

run 'echo "pass out on vr2 from vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'
}
if ! $isp1
set-state noisp1
if ! $isp2
set-state noisp2
}

state noisp1 {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'

run 'route change default 203.0.113.200'

run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr2 203.0.113.200) }" | pfctl -a loadbalance -f -'

run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'

run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'

run 'echo "pass out on vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'
}
if $isp1
set-state allworking
if ! $isp2
set-state alldown
}

state noisp2 {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'

run 'route change default 198.51.100.100'

run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr0 198.51.100.100) }" | pfctl -a loadbalance -f -'

run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr0 198.51.100.100)" | pfctl -a applications -f -'

run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'

run 'echo "pass out on vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'
}
if ! $isp1
set-state alldown
if $isp2
set-state allworking
}

state alldown {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
}
if $isp1 && ! $isp2
set-state noisp2
if $isp2 && ! $isp1
set-state noisp1
if $isp1 && $isp2
set-state all working
}

As ifstated is initialised & when it switches states, it flushes the anchors in the pf.conf, sets the default gateway so the host itself can be reachable remotely on the WAN and then injects rules into the PF anchors.

1 Comment

  1. This is a nice guide showing the basics of ifstated. I’ve been using ifstated for many years now and I use it from carp monitoring and failover, to ISP’s links failover and automatically switching. I’ve moved away from using a pf only based setup, to using multipath routing. This way the OpenBSD firewall can have more than one gateway and you can configure all of your isp’s gateways on it. With this setup, the connections originated from the firewall itself are also balanced across the links, and you can have a proxy server on it, instead of needing a dedicated machine for it. Also, you can have a dns server on it and the requests will be balanced across the links as well. The I use pf route-to and reply-to to direct the traffic for the machines behind the firewall and also requests from the internet. As for the timeouts, instead of removing the default routes, I play with route priorities and just add a higher priority route to the link(s) that is working. You can also have ifstated to run a pfctl -k to kill the states of the machines, but I’ve found that this is too aggressive. There are some few times that the link does recover and the connection do not fail, so keeping the states is good. But it all depends on your case. On the link failure detection, I only use ping to the internet as last resort. When the modem/router has snmp, I use it to detect the link failure. Also, some of them can be queried through their telnet or web interfaces. This way you don’t need to rely on an external server being on. But if you need to ping a external ip, I recommend that you don’t ping just one ip, but at least 2. That will prevent the case were the link is working, but the ping to the external ip failed because it was not working.

    Cheers

    Comment by Giancarlo Razzolini — 31/07/2014 @ 10:15 pm

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress