SIP is an all UDP protocol for VoIP (voice over IP) telephony.

Joao Filipe Placido, Jul 19, 2005

I have read some posts about a SIP module for lvs. Is this available? Does it do SIP dialog persistence?

Horms

Unfortunately it is not available as the project was halted before it became ready for release. I'm not entirely sure what SIP dialog persistance is, but in a nutshell all my design did was to provide persistance based on the Caller-ID, rather than the client's address and port as LVS ususally does.

Mike Machado mmachado (at) o1 (dot) com 20 Aug 2004

I have a LVS-DR setup with two realservers. My service is a voip application, SIP specifically. I am trying to balance requests between the two. I have all the LVS stuff setup, and was able to get the telnet test to work properly. With UDP though, there seems to be a problem. When the application is forming its reply, it uses the realserver as the source IP, instead of the VIP, as it does with the telnet test. I assume this is because UDP is stateless. I tried to SNAT the packets back to the correct IP, but you cannot SNAT locally generated packets.

I was able to change my voip application to just BIND to the VIP, but due to the nature of this application, it needs to be able to communicate on both the VIP and the RIP, I just want reply packets to use the same source IP and the inbound packets.

Anyone come across this problem for UDP applications, along with a possible solution?

Julian

What about using IP_PKTINFO in sendmsg, "srcip" is your server IP used in each request packet as daddr:e.g. example (http://www.ussg.iu.edu/hypermail/linux/kernel/0406.1/0771.html); thread (http://www.ussg.iu.edu/hypermail/linux/kernel/0406.1/index.html#0247).

Horms

Fixing the application to send reply packets from the addresses that they were received on is the best solution IMHO. It is the way I have resolved this problem in the past. The alternative would be to use LVS-NAT instead.

Erik Versaevel erik (at) infopact (dot) nl 13 Jan 2005

I'm currently trying to create a loadbalacing SIP (voip protocol) cluster, however for this to work I need SIP messages from the same call (identifiable by the sip callid field) to get to the same realserver over and over again. (so, I need persistence based on the contents of the SIP Call-ID field). This would call for ktcpvs as we need to process packets at layer 7, however that poses 2 new problems, the first is that SIP uses clear text UDP messages, not tcp and the second is that there are no SIP modules for ktcpvs.

Another option would be to mark SIP packets with iptables/netfilter based on the callid, however i run into the same problem, there are no modules to accomplish this.

I know that there are commercial products available who are able to do SIP session persistence based on callid, the F5 Big-IP for example, the downside of that is it costs around $ 10.000 for a single loadbalancer (which is a SPOF so you need 2) and is a bit overkill as i don't need multi gigabit loadbalancing.

High persistence won't work because reply packets from another SIP source might be balanced to the wrong server, ie packets from 192.168.0.2 might be balanced to real server 1, which sends it's reply directly to 192.168.0.3 (the end point for the call) but the replies from 192.168.0.3 might end up at another real server. (be aware I'm using direct routing because of NAT traversal)

Using TCP Sip would only solve half the problem (and couse some more). Answers to request could still end up at the wrong server (but one would only have to write a module for that, and not a kudpvs) and not all clients support TCP based sip.

Wensong Zhang wensong (at) linux-vs (dot) org 17 Jan 2005

We cannot use ktcpvs, because ktcpvs supports TCP only, and there is no SIP modules for TCP transport. The firewall marking doesn't solve the problem. However, we can write a special SIP UDP scheduling module for IPVS. It can detect the Call-Id from UDP packet and send it to the SIP server according to the recorded Call-Id table. We assume that there is no UDP fragments. Some NAT boxes (such as early IOS version of Cisco router) may drop UDP fragements except the first one.

Erik Versaevel

Such a module would definitly solve the problem. A round robin with call-id persistence would be awsome. Currently a device which can do that costs around $ 10000 each (times 2 for HA).

Malcolm Turnbull wrote:

If I wanted to test load balancing SIP using standard LVS UDP is their an OpenSource or Commercial Free Server to test against ?

Joe

AFAIK you can do all of VOIP PBX on Linux Asterisk The handsets (phones) are more of a problem. I believe you can get a linux box with a sound card and mic/headphones to be a phone. There was a large purchase of VOIP phones a while ago, that someone got to run linux, but these have all gone. You'll probably need to buy a real VOIP phone to test with.

Curt Moore tgrman21 (at) gmail (dot) com 11 Feb 2005

I believe that Wensong has the right idea here. Although SIP does support TCP the vast majority of SIP endpoints ony support UDP.

Asterisk can be used for a SIP application server but it's not geared/written to be a SIP proxy. For this you need something like SER, SIP Express Router.

When correctly configured, SER acting as a SIP proxy does support the distributing of calls to Asterisk boxes acting as media/application servers. The issue becomes how to load balance/distribute calls to multiple SER boxes, based on SIP call-id, so that the same SIP call-id always goes to the same SER box. Once you've statefully routed a call, based on call-id, to a particular SER box, SER can take over and ensure that things go to the correct Asterisk media/application server or endpoint based on its routing configuration. The director nodes just need to be smart enough to send the right call to the right SIP proxy residing in the LVS cluster.

As far as NAT goes, I've found though lots of experience that you'll never be able to penetrate every NAT implementation out there when it comes to SIP/UDP, you can only hope to get 99% of them as many of them don't fully conform to the RFC.

Benjamin Lawetz blawetz (at) teliphone (dot) ca 30 Jun 2005

I have a setup that load balances SIP UDP packets between 4 servers. Today one of my servers failed and mon removed it from the load-balancing, but some of the connections still remain and keep getting refreshed. I noticed something bizarre though with the ipvsadm UDP timeout, it is set to 35s, but the 3 connections that "stay stuck" seem to have an expire of 60 seconds instead of 35. And even though the server is removed and the rest of the connections timeout and get redirected to another server. Those 3 just keep going to the failed server.

Anyone have any idea why these 3 connections are (so it seems) auto-refreshing every 60s on a server that doesn't exist? Any way to clear these?

Ipvsadm startup script:

echo 1 > /proc/sys/net/ipv4/ip_forward /sbin/ipvsadm --set 0 0 35 /sbin/ipvsadm -A -u 192.168.89.220:5060 -s rr /sbin/ipvsadm -a -u 192.168.89.220:5060 -r 192.168.89.231 -i -w 5 /sbin/ipvsadm -a -u 192.168.89.220:5060 -r 192.168.89.232 -i -w 5 /sbin/ipvsadm -a -u 192.168.89.220:5060 -r 192.168.89.233 -i -w 5 /sbin/ipvsadm -a -u 192.168.89.220:5060 -r 192.168.89.234 -i -w 5

Before removal this is what I have:

Ipvsadm -L -n: IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn UDP 192.168.89.220:5060 rr -> 192.168.89.233:5060 Tunnel 5 0 4 -> 192.168.89.231:5060 Tunnel 5 0 3 -> 192.168.89.232:5060 Tunnel 5 0 3 -> 192.168.89.234:5060 Tunnel 5 0 6 Ipvsadm -L -n -c: IPVS connection entries pro expire state source virtual destination UDP 00:30 UDP 10.10.250.209:5060 192.168.89.220:5060 192.168.89.231:5060 UDP 00:25 UDP 192.168.85.25:1035 192.168.89.220:5060 192.168.89.233:5060 UDP 00:34 UDP 10.10.125.23:5060 192.168.89.220:5060 192.168.89.232:5060 UDP 00:26 UDP 206.55.81.128:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:22 UDP 10.10.233.189:5060 192.168.89.220:5060 192.168.89.232:5060 UDP 00:32 UDP 192.168.85.25:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:21 UDP 10.10.142.148:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:20 UDP 192.168.85.25:1036 192.168.89.220:5060 192.168.89.232:5060 UDP 00:34 UDP 10.10.249.211:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:23 UDP 10.10.249.211:1024 192.168.89.220:5060 192.168.89.233:5060 UDP 00:31 UDP 10.10.208.97:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:32 UDP 10.10.184.83:5060 192.168.89.220:5060 192.168.89.231:5060 UDP 00:27 UDP 70.80.53.141:5060 192.168.89.220:5060 192.168.89.233:5060 UDP 00:40 UDP 192.168.85.5:58040 192.168.89.220:5060 192.168.89.233:5060 UDP 00:28 UDP 10.10.14.83:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:25 UDP 10.10.215.175:6084 192.168.89.220:5060 192.168.89.231:5060

After removal I have:

Ipvsadm -L -n: IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn UDP 192.168.89.220:5060 rr -> 192.168.89.231:5060 Tunnel 5 0 3 -> 192.168.89.232:5060 Tunnel 5 0 3 -> 192.168.89.234:5060 Tunnel 5 0 6 Ipvsadm -L -n -c: IPVS connection entries pro expire state source virtual destination UDP 00:28 UDP 10.10.250.209:5060 192.168.89.220:5060 192.168.89.231:5060 UDP 00:31 UDP 192.168.85.25:1035 192.168.89.220:5060 192.168.89.231:5060 UDP 00:31 UDP 10.10.125.23:5060 192.168.89.220:5060 192.168.89.232:5060 UDP 00:31 UDP 10.10.81.128:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:29 UDP 10.10.233.189:5060 192.168.89.220:5060 192.168.89.232:5060 UDP 00:27 UDP 192.168.85.25:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:24 UDP 10.10.142.148:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:24 UDP 192.168.85.25:1036 192.168.89.220:5060 192.168.89.232:5060 UDP 00:33 UDP 10.10.249.211:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:40 UDP 10.10.249.211:1024 192.168.89.220:5060 192.168.89.233:5060 UDP 00:11 UDP 10.10.208.97:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:57 UDP 10.10.53.141:5060 192.168.89.220:5060 192.168.89.233:5060 UDP 00:57 UDP 192.168.85.5:58040 192.168.89.220:5060 192.168.89.233:5060 UDP 00:33 UDP 10.10.14.83:5060 192.168.89.220:5060 192.168.89.234:5060 UDP 00:33 UDP 10.10.215.175:6084 192.168.89.220:5060 192.168.89.231:5060

Sorry to answer myself, but narrowed down the problem.

I sniffed the traffic on the load balancer coming from one of the "stuck" IPs and I'm getting no traffic whatsoever. So basically while getting no traffic coming in, having no destination to go to and having a timeout of 35 seconds, the connection entries counts down from 59 seconds to 0 seconds and loops back to 59 seconds again. Is there a way to remove these connections? Anyone have any idea why they are looping when no traffic is coming in?

Horms

Which kernel do you have? This sounds like a counter bug that was resolved recently. However I couldn't convince myself that it manifests in 2.4 (I looked at 2.4.27). http://archive.linuxvirtualserver.org/html/lvs-users/2005-05/msg00043.html Also, I may as well mention this as it has been floating around: http://oss.sgi.com/archives/netdev/2005-06/msg00564.html

Marcos Hack marcoshack (at) gmail (dot) com 20 Dec 2005

I'm using LVS to load balance SIP UDP connections, and when a virtual service fail on a real server the director don't clean up the UDP connection on connection table. The solution seems to be just setting /proc/sys/net/ipv4/vs/expire_nodest_conn = 1 and change keepalived to remove the virtual service on failure instead of using "inhibit_on_failure" (set weight to 0).

Note

Kerberos is a secure authentication protocol. Many ports are involved, making it difficult to setup firewalls: e.g. Configuring your firewall to work with Kerberos. Further down this section, someone is ssh tunneling by LVS forwarding ssh to the realservers which use kerberos for authentication. In this case kerberos is just a login protocol that has nothing to do with LVS. There should only be one kerberos (ticket) server in your realm. You shouldn't LVS kerberos servers. Kerberos is a login protocol. There's no more reason to LVS kerberos, than there is to LVS login. However you can have the client login to kerberos'ed realservers.

several people

I'd be interested to know if LVS can be used and setup for silent login(no password prompting, i.e. using ticket forwarding) using ssh and kerberos.

Ryan Leathers ryan (dot) leathers (at) globalknowledge (dot) com 07 Mar 2006

this is not a good idea.

The kerberos replication system only permits one active admin server, so there is no opportunity for load balancing of the admin function. You shouldn't try to fool it by using LVS. You'll likely screw up the replication.

What you should do instead is to list multiple Kerberos key distribution centers (KDC'si) in your krb5.conf. Take a look at that and notice the section under [realms]. All you need to do is list multiple kdc's like so:

[realms]
	EXAMPLE.COM = {
	default_domain = example.com
	admin_server = krbadmin
	kdc = kdcserver1
	kdc = kdcserver2
	kdc = kdcserver3
	}

So, there you have it. By specifying multiple KDC servers you will be getting the behavior you really want, in case you lose a server for a little while, things will just keep on chugin' in your network. Don't get me wrong - LVS is a great tool, but its not the answer to every problem of service redundancy.

Ryan Leathers ryan (dot) leathers (at) globalknowledge (dot) com 02 Mar 2006

If you kerberize your host - that is to say, you have stuff like your ssh client using kerberos-compatible versions, and your authentication happening against kerberos, then kerberos works something like this:

I sit down at my favorite Linux workstation and open a shell so I can ssh to some other host on my network. Assuming I have not done so recently, the host I'm trying to reach won't be able to verify me. Instead, both my local host and the target host will rely on an authentication server to generate a new encryption key and distribute it to both parties. This is the session key. A Kerberos ticket is used to distribute the session key which includes info about me / my host that will be used by the target host to verify my connection. When the server passes this back to me, I forward it on to the target host as part of my authentication request. So, the ticket will be encrypted in a server key, which is known only by the server and the target host. nifty huh?

The ticket-granting-ticket just extends this to make life a little easier. We assume that some period of time is an acceptable amount for ticket granting without requiring the user to type in a password every time a ticket is needed. In short, I authenticate myself once to the server, and it allows me to perform any number of permitted authentications during the allowed time period. The ports used are likely going to be 88 for the kdc and possibly 749 for the admin server.

Karen Shepelak shepelak (at) fnal (dot) gov 04 Feb 2005

We are trying to get kerberos to work with LVS. We're ssh tunnelling to the realservers in an LVS forwarding ssh. We can kerberos through the ssh tunnel when connecting directly to the realservers, but not when we ssh to VIP:ssh. We get the following errors.

[karen@neptune karen]$ ssh -l shepelak minos-lvs01 Last login: Fri Feb 4 16:34:23 2005 from linux-test.fnal.gov aklog: Couldn't get fnal.gov AFS tickets: aklog: unknown RPC error (-1765328346) while getting AFS tickets /usr/X11R6/bin/xauth: timeout in locking authority file /afs/fnal.gov/files/home/room3/shepelak/.Xauthority Terminal type is xterm There are no available articles. /bin/touch: creating `/afs/fnal.gov/files/home/room3/shepelak/.Info': Permission denied <minos09>

Horms

We haven't got ldap working with LVS yet.

Konrads (dot) Smelkovs (at) emicovero (dot) com 03 Jul 2003

I have an LVS-DR, wlc, with three realservers running openldap 2.0. I have noticed that when going through the loadbalancer the nodes do not always reply, while if issuing requests directly, I get a reply all the time. The service is pretty busy (about 1k connections at any given moment). Adding persistence does not help. If I understand it correctly, LDAP is a simple TCP service. Usually, after performing an initial connection and query it idles there and reuses the connection.

Joe

Not knowing anything about ldap I looked at some of the HOWTOs. They are all long on configuring and using ldap, but none describe the ldap protocol. From the LDAP HOWTO I see that ldap uses iterative calls to the slapd (somewhat like DNS), that can be sent to other slapds (I don't know how this would work in your case). As well the slapd needs a 3rd tier database (eg dbm). If the clients are writing to the database, then you're going to have the many reader, many writer LVS problem. You're going to have to have only one copy of the database if clients are writing to the slapd

My setup: I have a "master" LDAP server that performs all writes and it replicates to the other "slave" servers. If a write request is sent to a slave, it is refered to the master server. The client then follows this reference and makes a succesful write. So in my case I have a stand-alone master, which is not load-balanced. Also, due to specific of the application (think authentification or such), it performs one or two exact search requests, like uid=konrads and requests for a attribute to be returned, e.g. : userPassword.

presumably these are two consecutive tcpip connections? If so you'll need persistence. As to what else is missing I don't know. Just for sanity checks, you can use these machines, IPs to setup some conventional LVS'ed service eg telnet, httpd? You know the ldap realservers are working OK outside of the LVS (ie you can connect to them directly, possibly after fiddling IPs)? After that it's brute force eg tcpdump I'm afraid. If you know the protocol well enough you can debug also with phatcat.

	Note
	Joe: multiport protocols are difficult to loadbalance under LVS

Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) indexmultimedia (dot) com 03/10/2006

Rmi is NOT a TCP protocol as is. It is a subprotocol that is similar to FTP. The two RMI ports are dealing a transaction on the standard RMI port, and then, there is a dynamic port negociation. There is no way to make RMI load balanced, as well as there is no way to make it go through a firewall...

If you look at google:RMI+Firewal, you will find relevant documents about port negociation and all the (bad) ways to handle this (bad) idea that is RMI.

Mr. CBoy cboy168 (at) gmail (dot) com 10 Mar 2006

Currently I am using LVS-Nat with 2 real JBoss servers. In my test environment I have 1 client that spawns hundreds of threads that will invoke methods through RMI. The flow is like this:

Client sends a request to the VIP for a naming proxy which will get forwarded to RS1. RS1 will then push down the naming proxy back to the client, but will be masquerated by the director. The client will ask for the stub on the remote object so that it can invoke methods and it will be returned. Now the client has the stub and can start to invoke methods, but here lies the problem. The stub came from RS1 and when the client calls a method on the stub, it gets load balanced to RS2 and becomes invalid. In a real world example, turning on persistance would remedy this however in a test environment, I'm limited by hardware. Is there anyway to get threads on the same client to talk to the same real server?

The client code is something simple like

MyObject obj = (MyObject) Naming.lookup("rmi://VIP/obj"); obj.callSomeMethod();

All the underlying connection details are handled by Sun's RMI so I'm unable to keep 1 connection for each thread myself, so my only guess is to figure out a way for LVS to handle it.