It’s possible to misuse NAT to load balance outbound traffic across multiple internet connections from different service providers,see the Load Balance Outgoing Traffic section of PF FAQ.
The shortfall with this configuration is when implemented alongside unstable links, forwarding will continue to be attempted over the links which are down, this will cause issues such as long hangs for users behind the NAT while connections time out. To mitigate this, ifstated can be used to smooth things over.
ifstated can be used to run tests & on event perform tasks, if you’re familiar with Cisco IOS, this is similar to some of what is available in EEM. In this scenario, ifstated will be set to ping each gateway at the service provider end of each link every 10 seconds & upon failure, adapt the configuration so traffic is not forwarded down that link. ifstated will continue to perform the tests & when tests start passing because link has re-established successfully, ifstated will reconfigure the system again so links are utilised.
For this post we’ll use the example ruleset from the PF FAQ and adapt it so it can be manipulated by ifstated.
Original pf.conf
lan_net = "192.168.0.0/24"
int_if = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "198.51.100.100"
ext_gw2 = "203.0.113.200"
# nat outgoing connections on each internet interface
match out on $ext_if1 from $lan_net nat-to ($ext_if1)
match out on $ext_if2 from $lan_net nat-to ($ext_if2)
# default deny
block in
block out
# pass all outgoing packets on internal interface
pass out on $int_if to $lan_net
# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
# load balance outgoing traffic from internal network.
pass in on $int_if from $lan_net \
route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } \
round-robin
# keep https traffic on a single connection; some web applications,
# especially "secure" ones, don't allow it to change mid-session
pass in on $int_if proto tcp from $lan_net to port https \
route-to ($ext_if1 $ext_gw1)
# general "pass out" rules for external interfaces
pass out on $ext_if1
pass out on $ext_if2
# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
pass out on $ext_if1 from $ext_if2 route-to ($ext_if2 $ext_gw2)
pass out on $ext_if2 from $ext_if1 route-to ($ext_if1 $ext_gw1)
Modified pf.conf
lan_net = "192.168.0.0/24"
int_if = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "198.51.100.100"
ext_gw2 = "203.0.113.200"
# nat outgoing connections on each internet interface
anchor nat-isp1
anchor nat-isp2
set skip on lo
# default deny
block in
block out
anchor "ftp-proxy/*"
# pass all outgoing packets on internal interface
pass out on $int_if to $lan_net
# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
# load balance outgoing traffic from internal network.
anchor loadbalance
# keep https traffic on a single connection; some web applications,
# especially "secure" ones, don't allow it to change mid-session
anchor applications
# general "pass out" rules for external interfaces
pass out on $ext_if1
pass out on $ext_if2
# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
anchor pass-isp1
anchor pass-isp2
The rules for NAT, load balancing & routing are replaced with anchors, ifstated will use these anchors to add & manipulate rules.
ifstated.conf
isp1 = '( "ping -q -c 1 -w 1 -S 198.51.100.199 198.51.100.100 >/dev/null" every 10)'
#If inteface is configured dynamically via dhcp use this instead
#isp2 = '( "ping -q -c 1 -w 1 -S `ifconfig vr2 inet |awk \'/inet/ { print $2 }\'` `awk \'/routers/ { print $3 }\' /var/db/dhclient.leases.vr2 |tail -1 |sed \'s/;//\'`>/dev/null" every 10)'
isp2 = '( "ping -q -c 1 -w 1 -S 203.0.113.220 203.0.113.200 >/dev/null" every 10)'
state allworking {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'route change default 203.0.113.200'
run 'echo "pass in on vr1 from 192.168.1.0/24 \
route-to { (vr0 198.51.100.100), (vr2 203.0.113.200) } round-robin" | pfctl -a loadbalance -f -'
run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'
run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'
run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'
run 'echo "pass out on vr0 from vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'
run 'echo "pass out on vr2 from vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'
}
if ! $isp1
set-state noisp1
if ! $isp2
set-state noisp2
}
state noisp1 {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
run 'route change default 203.0.113.200'
run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr2 203.0.113.200) }" | pfctl -a loadbalance -f -'
run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr2 203.0.113.200)" | pfctl -a applications -f -'
run 'echo "match out on vr2 from 192.168.1.0/24 nat-to (vr2)" | pfctl -a nat-isp2 -f -'
run 'echo "pass out on vr2 route-to (vr2 203.0.113.200)" | pfctl -a pass-isp2 -f -'
}
if $isp1
set-state allworking
if ! $isp2
set-state alldown
}
state noisp2 {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
run 'route change default 198.51.100.100'
run 'echo "pass in on vr1 from 192.168.1.0/24 route-to { (vr0 198.51.100.100) }" | pfctl -a loadbalance -f -'
run 'echo "pass in on vr1 proto tcp from 192.168.1.0/24 to port https route-to (vr0 198.51.100.100)" | pfctl -a applications -f -'
run 'echo "match out on vr0 from 192.168.1.0/24 nat-to (vr0)" | pfctl -a nat-isp1 -f -'
run 'echo "pass out on vr0 route-to (vr0 198.51.100.100)" | pfctl -a pass-isp1 -f -'
}
if ! $isp1
set-state alldown
if $isp2
set-state allworking
}
state alldown {
init {
run 'pfctl -a loadbalance -F rules'
run 'pfctl -a applications -F rules'
run 'pfctl -a nat-isp1 -F rules'
run 'pfctl -a nat-isp2 -F rules'
run 'pfctl -a pass-isp2 -F rules'
run 'pfctl -a pass-isp1 -F rules'
}
if $isp1 && ! $isp2
set-state noisp2
if $isp2 && ! $isp1
set-state noisp1
if $isp1 && $isp2
set-state all working
}
As ifstated is initialised & when it switches states, it flushes the anchors in the pf.conf, sets the default gateway so the host itself can be reachable remotely on the WAN and then injects rules into the PF anchors.