Shorewall Troubleshooting Guide

Tom Eastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.

2005-03-22



Table of Contents

First Steps
Check the FAQs.
Check the Errata
Try Searching the Shorewall Site and Mailing List Archives
shorewall start” and “shorewall restart” Errors
Some Things to Keep in Mind
Your Network Environment
New Device Doesn't Work?
Connection Problems
Ping Problems
Other Gotchas
Still Having Problems?
1. Revision History

First Steps

Some problems are easily solved by checking one of the resources described in the following sections.

Check the FAQs.

Check the FAQs for solutions to over 30 common problems.

Check the Errata

Check the Shorewall Errata to be sure that there isn't an update that you are missing for your version of the firewall.

Try Searching the Shorewall Site and Mailing List Archives

The Site and Mailing List Archives search facility can locate documents and posts about similar problems.

shorewall start” and “shorewall restart” Errors

If you receive an error message when starting or restarting the firewall and you can't determine the cause, then do the following:

  • Make a note of the error message that you see.

  • shorewall debug start 2> /tmp/trace

  • Look at the /tmp/trace file and see if that helps you determine what the problem is. Be sure you find the place in the log where the error message you saw is generated -- If you are using Shorewall 1.4.0 or later, you should find the message near the end of the log.

  • If you still can't determine what's wrong then see the support page.

Example 1. Startup Error

During startup, a user sees the following:

Adding Common Rules
iptables: No chain/target/match by that name
Terminated

A search through the trace for “No chain/target/match by that name” turned up the following:

+ echo 'Adding Common Rules'
+ add_common_rules
+ run_iptables -A reject -p tcp -j REJECT --reject-with tcp-reset
++ echo -A reject -p tcp -j REJECT --reject-with tcp-reset
++ sed 's/!/! /g'
+ iptables -A reject -p tcp -j REJECT --reject-with tcp-reset
iptables: No chain/target/match by that name

The command that failed was: “iptables -A reject -p tcp -j REJECT --reject-with tcp-reset”. In this case, the user had compiled his own kernel and had forgotten to include REJECT target support (see kernel.htm)

Some Things to Keep in Mind

  • You cannot test your firewall from the inside. Just because you send requests to your firewall external IP address does not mean that the request will be associated with the external interface or the “net” zone. Any traffic that you generate from the local network will be associated with your local interface and will be treated as loc->fw traffic.

  • IP addresses are properties of systems, not of interfaces. It is a mistake to believe that your firewall is able to forward packets just because you can ping the IP address of all of the firewall's interfaces from the local network. The only conclusion you can draw from such pinging success is that the link between the local system and the firewall works and that you probably have the local system's default gateway set correctly.

  • All IP addresses configured on firewall interfaces are in the $FW (fw) zone. If 192.168.1.254 is the IP address of your internal interface then you can write “$FW:192.168.1.254” in a rule but you may not write “loc:192.168.1.254”. Similarly, it is nonsensical to add 192.168.1.254 to the loc zone using an entry in /etc/shorewall/hosts.

  • Reply packets do NOT automatically follow the reverse path of the one taken by the original request. All packets are routed according to the routing table of the host at each step of the way. This issue commonly comes up when people install a Shorewall firewall parallel to an existing gateway and try to use DNAT through Shorewall without changing the default gateway of the system receiving the forwarded requests. Requests come in through the Shorewall firewall where the destination IP address gets rewritten but replies go out unmodified through the old gateway.

  • Shorewall itself has no notion of inside or outside. These concepts are embodied in how Shorewall is configured.

Your Network Environment

Many times when people have problems with Shorewall, the problem is actually an ill-conceived network setup. Here are several popular snafus:

  • Port Forwarding where client and server are in the same subnet. See FAQ 2.

  • Changing the IP address of a local system to be in the external subnet, thinking that Shorewall will suddenly believe that the system is in the “net” zone.

  • Multiple interfaces connected to the same HUB or Switch. Given the way that the Linux kernel respond to ARP “who-has” requests, this type of setup does NOT work the way that you expect it to. If you are running Shorewall version 1.4.7 or later, you can test using this kind of configuration if you specify the arp_filter option in /etc/shorewall/interfaces for all interfaces connected to the common hub/switch. Using such a setup with a production firewall is strongly recommended against.

New Device Doesn't Work?

If you have just added a new device such as VOIP and it doesn't work, be sure that you have assigned it an IP address in your local network and that its default gateway has been set to the IP address of your internal interface. For many of these devices, the simplest solution is to run a DHCP server; running it on your firewall is fine — be sure to set the dhcp option on your internal interface in /etc/shorewall/interfaces.

Connection Problems

If the appropriate policy for the connection that you are trying to make is ACCEPT, please DO NOT ADD ADDITIONAL ACCEPT RULES TRYING TO MAKE IT WORK. Such additional rules will NEVER make it work, they add clutter to your rule set and they represent a big security hole in the event that you forget to remove them later.

I also recommend against setting all of your policies to ACCEPT in an effort to make something work. That robs you of one of your best diagnostic tools - the “Shorewall” messages that Netfilter will generate when you try to connect in a way that isn't permitted by your rule set.

Check your log (“/sbin/shorewall show log”). If you don't see Shorewall messages, then your problem is probably NOT a Shorewall problem. If you DO see packet messages, it may be an indication that you are missing one or more rules -- see FAQ 17.

While you are troubleshooting, it is a good idea to clear two variables in /etc/shorewall/shorewall.conf:

LOGRATE=
LOGBURST=""

This way, you will see all of the log messages being generated (be sure to restart shorewall after clearing these variables).

Example 2. Log Message

Jun 27 15:37:56 gateway kernel: Shorewall:all2all:REJECT:IN=eth2
                                OUT=eth1 SRC=192.168.2.2
                                DST=192.168.1.3 LEN=67 TOS=0x00
                                PREC=0x00 TTL=63 ID=5805 DF
                                PROTO=UDP SPT=1803 DPT=53 LEN=47

Let's look at the important parts of this message:

  • all2all:REJECT - This packet was REJECTed out of the all2all chain -- the packet was rejected under the “all”->“all” REJECT policy (see FAQ 17).

  • IN=eth2 - the packet entered the firewall via eth2

  • OUT=eth1 - if accepted, the packet would be sent on eth1

  • SRC=192.168.2.2 - the packet was sent by 192.168.2.2

  • DST=192.168.1.3 - the packet is destined for 192.168.1.3

  • PROTO=UDP - UDP Protocol

  • DPT=53 - DNS

In this case, 192.168.2.2 was in the “dmz” zone and 192.168.1.3 is in the “loc” zone. I was missing the rule:

#ACTION   SOURCE           DEST                  PROTO   DEST
#                                                        PORT(S)
ACCEPT    dmz              loc                   udp     53

Ping Problems

Either can't ping when you think you should be able to or are able to ping when you think that you shouldn't be allowed? Shorewall's “Ping” Management is described here. Here are a couple of tips:

  • Remember that Shorewall doesn't automatically allow ICMP type 8 (“ping”) requests to be sent between zones. If you want pings to be allowed between zones, you need a rule of the form:

    #ACTION  SOURCE          DEST                  PROTO   DEST
    #                                                      PORT(S)
    AllowPing <source zone>   <destination zone>

    The ramifications of this can be subtle. For example, if you have the following in /etc/shorewall/nat:

    #EXTERNAL   INTERFACE  INTERNAL
    10.1.1.2    eth0       130.252.100.18

    and you ping 130.252.100.18, unless you have allowed icmp type 8 between the zone containing the system you are pinging from and the zone containing 10.1.1.2, the ping requests will be dropped.

  • Ping requests are subject to logging under your policies. So ping floods can cause an equally big flood of log messages. To eliminate these, as the last rule in your /etc/shorewall/rules file add:

    #ACTION  SOURCE          DEST                  PROTO   DEST
    #                                                      PORT(S)
    DropPing net             all

Other Gotchas

  • Seeing rejected/dropped packets logged out of the INPUT or FORWARD chains? This means that:

    1. your zone definitions are screwed up and the host that is sending the packets or the destination host isn't in any zone (using an /etc/shorewall/hosts file are you?); or

    2. the source and destination hosts are both connected to the same interface and you don't have a policy or rule for the source zone to or from the destination zone or you haven't set the routeback option for the interface in /etc/shorewall/interfaces.

  • If you specify “routefilter” for an interface, that interface must be up prior to starting the firewall.

  • Is your routing correct? For example, internal systems usually need to be configured with their default gateway set to the IP address of their nearest firewall interface. One often overlooked aspect of routing is that in order for two hosts to communicate, the routing between them must be set up in both directions. So when setting up routing between A and B, be sure to verify that the route from B back to A is defined.

  • Some versions of LRP (EigerStein2Beta for example) have a shell with broken variable expansion. You can get a corrected shell from the Shorewall Errata download site.

  • Do you have your kernel properly configured? Click here to see my kernel configuration.

  • Shorewall requires the “ip” program. That program is generally included in the “iproute” package which should be included with your distribution (though many distributions don't install iproute by default). You may also download the latest source tarball from http://developer.osdl.org/dev/iproute2/download/ .

  • Problems with NAT? Be sure that you let Shorewall add all external addresses to be use with NAT unless you have set ADD_IP_ALIASES =No in /etc/shorewall/shorewall.conf.

Still Having Problems?

See the Shorewall Support Page.

1. Revision History

Revision History
Revision 1.92004-08-25TE
Advice for the networking-challenged.
Revision 1.82004-04-03TE
Point out that firewall addresses are in the $FW zone.
Revision 1.72004-02-02TE
Add hint about testing from inside the firewall.
Revision 1.62004-01-06TE
Add pointer to Site and Mailing List Archives Searches.
Revision 1.52004-01-01TE
Added information about eliminating ping-generated log messages.
Revision 1.42003-12-22TE
Initial Docbook Conversion