Copyright © 2009 Thomas M. Eastep
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.
2009/08/09
Table of Contents
- Introduction
- Network Topology
- Shorewall Configuration
- /etc/shorewall/params
- /etc/shorewall/shorewall.conf
- /etc/shorewall/actions
- /etc/shorewall/action.Mirrors
- /etc/shorewall/zones
- /etc/shorewall/interfaces
- /etc/shorewall/hosts
- /etc/shorewall/policy
- /etc/shorewall/accounting
- /etc/shorewall/blacklist
- /etc/shorewall/compile
- /etc/shorewall/findgw
- /etc/shorewall/isusable
- /etc/shorewall/lib.private
- /etc/shorewall/masq
- /etc/shorewall/notrack
- /etc/shorewall/providers
- /etc/shorewall/proxyarp
- /etc/shorewall/restored
- /etc/shorewall/route_rules
- /etc/shorewall/routestopped
- /etc/shorewall/rules
- /etc/shorewall/started
- /etc/shorewall/stopped
- /etc/shorewall/tcdevices
- /etc/shorewall/tcclasses
- /etc/shorewall/tcfilters
- /etc/shorewall/tcrules
- /etc/shorewall/tunnels
Caution
The ruleset shown in this article uses Shorewall features that are not available in Shorewall versions prior to 4.4.0.
The configuration described in this article represents the network at shorewall.net during the summer of 2009. It uses the following Shorewall features:
A DMZ with two "systems" using Proxy ARP and running in OpenVZ Virtual Environments
Linux runs the firewall and the servers (although they run in OpenVZ containers on the firewall system). Linux is not used natively on any of our other systems except for an HP mini which runs HP Mobile Internet Experience (MIE) -- essentially Ubuntu Hardy. I rather run Windows natively (either Vista Home Premium or XP Professional) and run Linux in VMs under VirtualBox. This approach has a number of advantages:
Efficient disk utilization.
The virtual disks used by Linux are just files in the NTFS file system. There is no need to pre-allocate one or more partitions for use by Linux. Some large applications, like Google Earth, are installed only on Windows.
Avoids proprietary hardware issues.
The Linux VMs emulate standard hardware that is well-supported by Linux.
Avoids DRM hassles
All DRM-protected media can be handled under Windows.
Websites that don't work with Firefox (or at least with Linux Firefox)
VirtualBox is fast (when your processor supports virtualization extensions) and very easy to use. I highly recommend it!
Our network is diagrammed in the following graphic.
We have accounts with two different ISPs:
Comcast
This is a high-speed (20mb/4mb) link with a single dynamic IPv4 address. We are not allowed to run servers accessible through this account.
Avvanta
This is a low-speec (1.5mb/384kbit) link with five static IP address. Our servers are accessed through this account.
The wired local network is restricted to my home office. The wireless network is managed by a Linksys WRT300N pre-N router which we use only as an access point -- its WAN interface is unused and it is configured to not do NAT. The wireless network uses WPA2 personal security and MAC filtering is enabled in the router. These two factors make it a hassle when guests visit with a laptop but provide good security for the network.
This section contains exerpts from the Shorewall configuration.
It is important to keep in mind that parts of my configuration are there just to provide a test bed for Shorewall features. So while they show correct usage, they don't necessarily provide any useful benefit. I have tried to point those out in the sub-sections that follow.
MIRRORS=62.216.169.37,\ 63.229.2.114,\ ... NTPSERVERS=... POPSERVERS=... LOG=ULOG INT_IF=eth1 EXT_IF=eth2 COM_IF=eth0 VPS_IF=venet0
As shown, this file defines variables to hold the various lists of IP addresses that I need to maintain. To simplify network reconfiguration, I also use variables to define the log level and the network interfaces.
###############################################################################
# S T A R T U P E N A B L E D
###############################################################################
STARTUP_ENABLED=Yes
###############################################################################
# V E R B O S I T Y
###############################################################################
VERBOSITY=0
###############################################################################
# C O M P I L E R
# (setting this to 'perl' requires installation of Shorewall-perl)
###############################################################################
SHOREWALL_COMPILER=perl
###############################################################################
# L O G G I N G
###############################################################################
LOGFILE=/var/log/ulog/syslogemu.log
STARTUP_LOG=/var/log/shorewall-init.log
LOG_VERBOSITY=2
LOGFORMAT="%s:%s:"
LOGTAGONLY=No
LOGRATE=
LOGBURST=
LOGALLNEW=
BLACKLIST_LOGLEVEL=
MACLIST_LOG_LEVEL=
TCP_FLAGS_LOG_LEVEL=$LOG
SMURF_LOG_LEVEL=$LOG
LOG_MARTIANS=No
###############################################################################
# L O C A T I O N O F F I L E S A N D D I R E C T O R I E S
###############################################################################
IPTABLES=
IPSET=
PATH=/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
SHOREWALL_SHELL=/bin/sh
SUBSYSLOCK=
MODULESDIR=
CONFIG_PATH=/etc/shorewall:/usr/share/shorewall
RESTOREFILE=
IPSECFILE=zones
LOCKFILE=
###############################################################################
# D E F A U L T A C T I O N S / M A C R O S
###############################################################################
DROP_DEFAULT="Drop"
REJECT_DEFAULT="Reject"
ACCEPT_DEFAULT="none"
QUEUE_DEFAULT="none"
###############################################################################
# R S H / R C P C O M M A N D S
###############################################################################
RSH_COMMAND='ssh ${root}@${system} ${command}'
RCP_COMMAND='scp ${files} ${root}@${system}:${destination}'
###############################################################################
# F I R E W A L L O P T I O N S
###############################################################################
IP_FORWARDING=Yes
ADD_IP_ALIASES=No
ADD_SNAT_ALIASES=No
RETAIN_ALIASES=No
TC_ENABLED=Internal
TC_EXPERT=No
CLEAR_TC=Yes
MARK_IN_FORWARD_CHAIN=Yes
CLAMPMSS=Yes
ROUTE_FILTER=No
DETECT_DNAT_IPADDRS=Yes
MUTEX_TIMEOUT=60
ADMINISABSENTMINDED=Yes
BLACKLISTNEWONLY=Yes
DELAYBLACKLISTLOAD=No
MODULE_SUFFIX=ko
DONT_LOAD=
DISABLE_IPV6=No
BRIDGING=No
DYNAMIC_ZONES=No
PKTTYPE=No
MACLIST_TABLE=mangle
MACLIST_TTL=60
SAVE_IPSETS=No
MAPOLDACTIONS=No
FASTACCEPT=No
IMPLICIT_CONTINUE=Yes
HIGH_ROUTE_MARKS=Yes
USE_ACTIONS=Yes
OPTIMIZE=1
EXPORTPARAMS=Yes
EXPAND_POLICIES=Yes
KEEP_RT_TABLES=No
DELETE_THEN_ADD=No
MULTICAST=Yes
AUTO_COMMENT=Yes
MANGLE_ENABLED=Yes
NULL_ROUTE_RFC1918=Yes
USE_DEFAULT_RT=No
RESTORE_DEFAULT_ROUTE=No
FAST_STOP=Yes
AUTOMAKE=No
LOG_MARTIANS=Yes
WIDE_TC_MARKS=Yes
###############################################################################
# P A C K E T D I S P O S I T I O N
###############################################################################
BLACKLIST_DISPOSITION=DROP
MACLIST_DISPOSITION=ACCEPT
TCP_FLAGS_DISPOSITION=DROP
I don't believe that there is anything remarkable there
#ACTION Mirrors # Accept traffic from Shorewall Mirrors
I make this into an action so the rather long list of rules go into their own chain.
#TARGET SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE # PORT PORT(S) DEST LIMIT COMMENT Accept traffic from Mirrors ACCEPT $MIRRORS
See the rules file -- this action is used for rsync traffic.
fw firewall loc ipv4 #Local Zone dmz ipv4 #DMZ net ipv4 #Internet vpn:loc,net ipsec #IPSEC drct:loc ipv4 #Direct internet access
The vpn zone is mostly for testing Shorewall IPSEC support. It is nested in loc and net to test a feature added in Shorewall 4.4.0. The drct zone is a dynamic zone whose members bypass the transparent proxy. Some applications (such as VirtualBox registration) don't work through the proxy.
#ZONE INTERFACE BROADCAST OPTIONS loc $INT_IF detect dhcp,logmartians=1,routefilter=1,tcpflags dmz $VPS_IF detect logmartians=1,routefilter=0,routeback net $EXT_IF detect dhcp,blacklist,tcpflags,optional,routefilter=0,nosmurfs,logmartians=0,proxyarp=1 net $COM_IF detect dhcp,blacklist,tcpflags,optional,upnp,routefilter=0,nosmurfs,logmartians=0 loc tun+ detect
Notice that VPN clients are treated the same as local hosts.
I set the proxyarp option on $EXT_IF so that
The firewall will respond to ARP who-has requests for the servers in the DMZ.
To keep OpenVZ happy (it issues dire warnings if the option is not set on the associated external interface).
#ZONE HOST(S) OPTIONS vpn $EXT_IF:0.0.0.0/0 vpn $COM_IF:0.0.0.0/0 vpn $INT_IF:0.0.0.0/0 drct $INT_IF:dynamic
The vpn zone includes ipsec hosts interfacing from either external interface as well as the local interface. drct is defined as dynamic through the local interface (recall that it is a sub-zone of loc).
#SOURCE DEST POLICY LOG LIMIT:BURST # LEVEL $FW dmz REJECT $LOG $FW all ACCEPT loc net ACCEPT - loc fw ACCEPT loc vpn ACCEPT vpn fw ACCEPT vpn loc ACCEPT net net NONE net all DROP $LOG 8/sec:30 dmz fw REJECT $LOG all fw DROP $LOG all all REJECT $LOG
I'm a bit sloppy with my fw<->loc policies -- I should fix that someday...
#ACTION CHAIN SOURCE DESTINATION PROTO DEST SOURCE USER/ # PORT(S) PORT(S) GROUP hp:COUNT accounting $COM_IF $INT_IF:172.20.1.107 UDP hp:COUNT accounting $INT_IF:172.20.1.107 $COM_IF UDP DONE hp mail:COUNT - $EXT_IF $VPS_IF:206.124.146.0/24 tcp 25 mail:COUNT - $VPS_IF:206.124.146.0/24 $EXT_IF tcp 25 DONE mail web - $EXT_IF $VPS_IF:206.124.146.0/24 tcp 80 web - $EXT_IF $VPS_IF:206.124.146.0/24 tcp 443 web - $VPS_IF:206.124.146.0/24 $EXT_IF tcp - 80 web - $VPS_IF:206.124.146.0/24 $EXT_IF tcp - 443 COUNT web $EXT_IF $VPS_IF:206.124.146.0/24 COUNT web $VPS_IF:206.124.146.0/24 $EXT_IF
The accounting chains are as follows:
hp
Counts traffic to/from my work laptop to HP. The VPN users NAT-Traversal (UDP 4500) so I just count all UDP traffic to/from my work system.
mail
Incoming and outgoing email
web
Website traffic (both HTTP and HTTPS)
#ADDRESS/SUBNET PROTOCOL PORT - udp 1024:1033,1434 - tcp 57,1433,1434,2401,2745,3127,3306,3410,4899,5554,6101,8081,9898
This configuration silently drops a few ports that get lots of traffic.
use strict; use Shorewall::Chains; my $chainref = ensure_manual_chain qw/DNS_DDoS/; add_rule $chainref, q(-m string --algo bm --from 30 --to 31 --hex-string "|010000010000000000000000020001|" -j DROP); add_rule $chainref, q(-m string --algo bm --from 30 --to 31 --hex-string "|000000010000000000000000020001|" -j DROP); add_rule $chainref, q(-j ACCEPT); 1;
The above was created during a recent DDOS incident that targeted DNS servers. It illustrates how manual chains can be created.
if [ -f /var/lib/dhcp3/dhclient.${1}.leases ]; then
grep 'option routers' /var/lib/dhcp3/dhclient.${1}.leases | tail -n 1 | while read j1 j2 gateway; do echo $gateway | sed 's/;//'; return 0; done
fiThe Comcast line has a dynamic IP address assigned with the help of dhclient.
local status
status=0
[ -f /etc/shorewall/${1}.status ] && status=$(cat /etc/shorewall/${1}.status)
return $statusFor use with lsm.
start_lsm() {
killall lsm 2> /dev/null
cat <<EOF > /etc/lsm/shorewall.conf
connection {
name=Avvanta
checkip=206.124.146.254
device=$EXT_IF
ttl=2
}
connection {
name=Comcast
checkip=${ETH0_GATEWAY:-71.231.152.1}
device=$COM_IF
ttl=1
}
EOF
rm -f /etc/shorewall/*.status
/usr/sbin/lsm /etc/lsm/lsm.conf >> /var/log/lsm
}
This function configures and starts lsm.
#INTERFACE SOURCE ADDRESS COMMENT Masquerade Local Network $COM_IF 0.0.0.0/0 $EXT_IF !206.124.146.0/24 206.124.146.179
All connections out through Comcast must have the dynamically assigned address as their source address. Traffic from hosts without an Avvanta public IP address get 206.124.146.179 as their source address.
#SOURCE DESTINATION PROTO DEST SOURCE USER/ # PORT(S) PORT(S) GROUP net:!192.88.99.1 - 41 dmz 206.124.146.255 udp dmz 255.255.255.255 udp loc 172.20.1.255 udp loc 255.255.255.255 udp $FW 255.255.255.255 udp $FW 172.20.1.255 udp $FW 206.124.146.255 udp
This file omits the 6to4 traffic originating from 6to4 relays as well as broadcast traffic (which Netfilter doesn't handle).
#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY Avvanta 1 0x10000 main $EXT_IF 206.124.146.254 track,loose,fallback $INT_IF,$VPS_IF,tun* Comcast 2 0x20000 main $COM_IF detect track,balance $INT_IF,$VPS_IF,tun*
See the Multi-ISP article for an explaination of the multi-ISP aspects of this configuration.
<empty>
As mentioned above, I set the proxyarp on the associated external interface instead of defining proxy ARP in this file.
if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
start_lsm
fi
chmod 744 ${VARDIR}/stateIf lsm isn't running then start it. Make the state file world-readable.
#SOURCE DEST PROVIDER PRIORITY - 172.20.0.0/24 main 1000 #OpenVPN clients - 206.124.146.177 main 1001 #Servers -- Routes configured by OpenVZ - 206.124.146.178 main 1001 # - 216.168.3.44 Avvanta 1001 #NNTP -- Does source IP verification 206.124.146.176/30 - Avvanta 26000 #Avvanta public IP addresses 206.124.146.180 - Avvanta 26000 #
These entries simply ensure that outgoing traffic uses the correct interface.
#INTERFACE HOST(S) OPTIONS PROTO $INT_IF 172.20.1.0/24 source,dest $VPS_IF 206.124.146.177,206.124.146.178 $EXT_IF - notrack 41
Keep the lights on while Shorewall is stopped.
###############################################################################################################################################################################
#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/
# PORT PORT(S) DEST LIMIT GROUP
###############################################################################################################################################################################
SECTION ESTABLISHED
SECTION RELATED
SECTION NEW
REJECT:$LOG loc net tcp 25 #Stop direct loc->net SMTP (Comcast uses submission).
REJECT:$LOG loc net udp 1025:1031 #MS Messaging
COMMENT Stop NETBIOS crap
REJECT loc net tcp 137,445
REJECT loc net udp 137:139
COMMENT Stop my idiotic work laptop from sending to the net with an HP source IP address
DROP loc:!172.20.0.0/23 net
COMMENT
###############################################################################################################################################################################
# Local Network to Firewall
#
NONAT drct -
REDIRECT- loc 3128 tcp 80 - !66.199.187.46,172.20.1.108,206.124.146.177,155.98.64.80,81.19.16.0/21
###############################################################################################################################################################################
# Local network to DMZ
#
ACCEPT loc dmz udp domain,177
ACCEPT loc dmz tcp ssh,smtp,465,587,www,ftp,imaps,domain,https,5901:5903 -
ACCEPT loc dmz udp 33434:33524
###############################################################################################################################################################################
# Internet to ALL -- drop NewNotSyn packets
#
dropNotSyn net fw tcp
dropNotSyn net loc tcp
dropNotSyn net dmz tcp
###############################################################################################################################################################################
# Internet to DMZ
#
DNS_DDoS net dmz udp domain
ACCEPT net dmz tcp smtp,www,ftp,465,587,imaps,domain,https
ACCEPT net dmz udp 33434:33454
Mirrors:none net dmz tcp 873
ACCEPT net dmz tcp 22 - - s:ssh:3/min:3
#############################################################################################################################################################
#################
#
# Net to Local
#
Limit:$LOG:SSHA,3,60\
net loc tcp 22
#
# BitTorrent from Wireless Network
#
#DNAT net:$COM_IF loc:172.20.1.102 tcp 6881:6889
#DNAT net:$COM_IF loc:172.20.1.102 udp 6881
#
# UPnP
#
forwardUPnP net loc
#
# Silently Handle common probes
#
REJECT net loc tcp www,ftp,https
DROP net loc icmp 8
###############################################################################################################################################################################
# DMZ to Internet
#
ACCEPT dmz net udp domain,ntp
REJECT dmz net:$COM_IF tcp smtp
ACCEPT dmz net tcp echo,ftp,ssh,smtp,whois,domain,www,81,nntp,https,2401,2702,2703,8080
ACCEPT dmz net:$POPSERVERS tcp pop3
#
# Some FTP clients seem prone to sending the PORT command split over two packets. This prevents the FTP connection tracking
# code from processing the command and setting up the proper expectation. The following rule allows active FTP to work in these cases
# but logs the connection so I can keep an eye on this potential security hole.
#
ACCEPT:$LOG dmz net tcp 1024: 20
###############################################################################################################################################################################
# DMZ to Local
#
ACCEPT dmz loc tcp 22 - - s:ssh:3/min:3
###############################################################################################################################################################################
# DMZ to Firewall -- ntp & snmp Silently reject Auth
#
ACCEPT dmz fw tcp 161,ssh
ACCEPT dmz fw udp 161,ntp
REJECT dmz fw tcp auth
###############################################################################################################################################################################
# Internet to Firewall
#
REJECT net fw tcp www,ftp,https
DROP net fw icmp 8
ACCEPT net fw udp 33434:33454
ACCEPT net fw tcp 22 - - s:ssh:3/min:3
ACCEPT net fw udp 33434:33524
###############################################################################################################################################################################
# Firewall to DMZ
#
ACCEPT fw dmz tcp domain,www,ftp,ssh,smtp,https,993,465,587,5901
ACCEPT fw dmz udp domain
REJECT fw dmz udp 137:139
##############################################################################################################################################################################
#
COMMENT Freenode Probes
DROP net:82.96.96.3,85.190.0.3 any
COMMENT
##############################################################################################################################################################################
# Allow Ping except where disallowed earlier
#
ACCEPT any any icmp 8if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
start_lsm
fi
chmod 744 ${VARDIR}/stateIf lsm isn't running then start it. Make the state file world-readable.
if [ "$COMMAND" = stop -o "$COMMAND" = clear ]; then
killall lsm 2> /dev/null
fi
chmod 744 ${VARDIR}/stateKill lsm if the command is stop or clear. Make the state file world-readable.
#INTERFACE IN-BANDWITH OUT-BANDWIDTH OPTIONS $EXT_IF - 300kbit classify $INT_IF - 80mbit classify $COM_IF - 4mbit classify,hfsc
The use of HFSC on the Comcast link is largely to provide a test bed for that qdisc; I really don't have any real-time requirement such as VOIP.
#INTERFACE MARK RATE CEIL PRIORITY OPTIONS 1:110 - full/4 full 1 tcp-ack,tos-minimize-delay 1:120 - full/4 full 2 flow=nfct-src 1:130 - full/4 230kbit 3 default,flow=nfct-src 1:140 - full/4 230kbit 4 flow=nfct-src 2:10 - 95*full/100 full 1 flow=dst 2:100 - 14mbit 20mbit 2 2:100:101 - 7mbit 20mbit 3 default,flow=dst 2:100:102 - 7mbit 20mbit 3 flow=dst 3:10 - 2mbit:4ms full 1 flow=nfct-src 3:100 - 2mbit full 2 3:100:101 - 1mbit full 3 default,flow=nfct-src 3:100:102 - 1mbit full 3 flow=nfct-src
Note that most of the outgoing bandwidth on the local interface is allocated to one class. That class is used for local traffic.
#INTERFACE: SOURCE DEST PROTO DEST SOURCE TOS LENGTH #CLASS PORT(S) PORT(S) # =============================== AVVANTA ==================================== # # Give Highest priority to LSM's pings to the gateway and to DNS queries # 1:110 206.124.146.176 206.124.146.254 icmp 1:110 206.124.146.177 - udp 53 # # Second Highest priority to IPv6 Tunnel # 1:120 206.124.146.180 # # Lowest priority to bulk traffic # 1:140 206.124.146.177 - tcp - 873 - 2048 1:140 206.124.146.177 - - - - tos-minimize-cost
The tcfilters file is only used for the Avvanta provider because it has static public IP addresses.
#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS # PORT(S) COMMENT Shape incoming traffic # # Most of the bandwidth is reserved for local traffic since the downlinks aren't that fast # 2:10 206.124.146.176/30 $INT_IF 2:10 206.124.146.177 $INT_IF 2:10 172.20.1.254 $INT_IF # # Guarantee 1/2 of the incoming bandwidth for my work system # 2:102 0.0.0.0/0 $INT_IF:172.20.1.107 COMMENT Shape outgoing traffic to Comcast # # Give 1/2 to my work system and add a latency guarantee # 3:10 172.20.1.107 $COM_IF # # Restrict Torrent uploads # 3:102 172.20.1.0/24 $COM_IF tcp - 6881:6889
The tcrules file is used to classify traffic that deals with the local network and/or with Comcast.