Troubleshoot Networking - OpenStack Cloud Administrator Guide

Cannot reach floating IPs

If you cannot reach your instances through the floating IP address, check the following:

Ensure the default security group allows ICMP (ping) and SSH (port 22), so that you can reach the instances:

$ nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port |  IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

Ensure the NAT rules have been added to iptables on the node that nova-network is running on, as root:

# iptables -L -nv -t nat
-A nova-network-PREROUTING -d 68.99.26.170/32 -j DNAT --to-destination 10.0.0.3
-A nova-network-floating-snat -s 10.0.0.3/32 -j SNAT --to-source 68.99.26.170

Check that the public address, in this example "68.99.26.170", has been added to your public interface. You should see the address in the listing when you enter "ip addr" at the command prompt.
```
$ ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether xx:xx:xx:17:4b:c2 brd ff:ff:ff:ff:ff:ff
inet 13.22.194.80/24 brd 13.22.194.255 scope global eth0
inet 68.99.26.170/32 scope global eth0
inet6 fe80::82b:2bf:fe1:4b2/64 scope link
valid_lft forever preferred_lft forever
```
Note that you cannot SSH to an instance with a public IP from within the same server as the routing configuration won't allow it.
You can use tcpdump to identify if packets are being routed to the inbound interface on the compute host. If the packets are reaching the compute hosts but the connection is failing, the issue may be that the packet is being dropped by reverse path filtering. Try disabling reverse-path filtering on the inbound interface. For example, if the inbound interface is eth2, as root, run:
```
# sysctl -w net.ipv4.conf.eth2.rp_filter=0
```
If this solves your issue, add the following line to /etc/sysctl.conf so that the reverse-path filter is disabled the next time the compute host reboots:
```
net.ipv4.conf.rp_filter=0
```

Disable firewall

To help debug networking issues with reaching VMs, you can disable the firewall by setting the following option in /etc/nova/nova.conf:

firewall_driver=nova.virt.firewall.NoopFirewallDriver

We strongly recommend you remove this line to re-enable the firewall once your networking issues have been resolved.

Packet loss from instances to nova-network server (VLANManager mode)

If you can SSH to your instances but you find that the network interactions to your instance is slow, or if you find that running certain operations are slower than they should be (for example, sudo), then there may be packet loss occurring on the connection to the instance.

Packet loss can be caused by Linux networking configuration settings related to bridges. Certain settings can cause packets to be dropped between the VLAN interface (for example, vlan100) and the associated bridge interface (for example, br100) on the host running the nova-network service.

One way to check whether this is the issue in your setup, is to open up three terminals and run the following commands:

In the first terminal, on the host running nova-network, use tcpdump on the VLAN interface to monitor DNS-related traffic (UDP, port 53). As root, run:
```
# tcpdump -K -p -i vlan100 -v -vv udp port 53
```
In the second terminal, also on the host running nova-network, use tcpdump to monitor DNS-related traffic on the bridge interface. As root, run:
```
# tcpdump -K -p -i br100 -v -vv udp port 53
```
In the third terminal, SSH inside of the instance and generate DNS requests by using the nslookup command:
```
$ nslookup www.google.com
```
The symptoms may be intermittent, so try running nslookup multiple times. If the network configuration is correct, the command should return immediately each time. If it is not functioning properly, the command hangs for several seconds.
If the nslookup command sometimes hangs, and there are packets that appear in the first terminal but not the second, then the problem may be due to filtering done on the bridges. Try to disable filtering, run the following commands as root:
```
# sysctl -w net.bridge.bridge-nf-call-arptables=0
# sysctl -w net.bridge.bridge-nf-call-iptables=0
# sysctl -w net.bridge.bridge-nf-call-ip6tables=0
```
If this solves your issue, add the following line to /etc/sysctl.conf so that these changes take effect the next time the host reboots:
```
net.bridge.bridge-nf-call-arptables=0
net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-ip6tables=0
```

KVM: Network connectivity works initially, then fails

Some administrators have observed an issue with the KVM hypervisor where instances running Ubuntu 12.04 sometimes loses network connectivity after functioning properly for a period of time. Some users have reported success with loading the vhost_net kernel module as a workaround for this issue (see bug #997978) . This kernel module may also improve network performance on KVM. To load the kernel module, as root:

# modprobe vhost_net

	Note
	Loading the module has no effect on running instances.