If you install sshd and generate the host keys for the realservers using the default settings, you'll get a working LVS'ed sshd. However you should be aware of what you've done. The default sshd listens to 0.0.0.0 and you will have generated host keys for a machine whose name corresponds to the RIP (and not the VIP). Since the client will be displaying a prompt with the name of the realserver (rather than the name associated with the VIP) this will work just fine. However the client will get a different realserver each connection (which is OK too) and will accumulate keys for each realserver. If instead you want the client to be presented with one virtual machine, you will need each machine to have its hostname being the name associated with the VIP, the sshd will have to listen to the VIP (if LVS-DR, LVS-Tun) and the hostkeys will have to be generated for the name of the VIP.

The user must exit cleanly from their ssh session, or the realserver will be left running the ssh invoked process at high load average. The problem is that what the user thinks is a clean exit and what the sshd thinks are a clean exit, may be different things. (There is a similar problem on a webserver, which is running a process invoked by a cgi script, when the client disconnects by clicking to another page or hitting "stop").

Shivaji Navale

On our realservers, after the users have logged out of their ssh session, zombie processes run at high load average in the background. We resort to killing the zombie processess. The ssh connection to the director doesnt get closed even after ctrl-D. From netstat, the connection is still active.

Dave Jagoda dj (at) opsware (dot) com 23 Nov 2003

Does this sound like what's going on? http://www.openssh.org/faq.html#3.10, ssh hangs on exit.

You do not need persistence with ssh, but you can use it.

Piero Calucci calucci (at) sissa (dot) it 30 Mar 2004

I use ipvs to load balance ssh and I do use persistence (and my users are happy with it -- in fact they asked me to do so). With this setup when they open multiple sessions they are guaranteed to reach always the same realserver, so they can see all their processes and local /tmp files.

If you use persistence, be aware of the effects.

jeremy (at) xxedgexx (dot) com

I'm using ipvs to balance ssh connections but for some reason ipvs is only using one real server and persists to use that server until I delete its arp entry from the ipvs machine and remove the virtual loopback on the realserver. Also, I noticed that connections do not register with this behavior.

Wensong

do you use the persistent port for your VIP:22? If so, the default timeout of persistent port is 360 seconds, once the ssh session finishes, it takes 360 seconds to expire the persistent session. (In ipvs-0.9.1, you can flexibly set the timeout for the persistent port.) There is no need to use persistent port for ssh service, because the RSA keys are exchanged in each ssh session, and each session is not related.

	Note
	Joe - if mail arrives on different realservers, then people have tried merging/synching the files on the different machines. Only one of the file formats, maildir, is worth attempting. Even that hasn't worked out too well and it seems that most successful LVS mail farms are using centralised file servers.

Todd Lyons tlyons (at) ivenue (dot) com 13 Jan 2006

I don't know that I'd feel all that comfortable about syncing either one: mbox - This could end up with a corrupted mbox file since it's all one big long file. maildir - Guaranteed that you won't end up with any filename conflicts since the hostname is used in the filename, but there are some shared files, such as the quota or subscribed files. Those files could easily get out of sync.

Graeme Fowler graeme (at) graemef (dot) net 13 Jan 2006

Indeed they can (says someone long since burnt, not exactly by rsync, but by some other methods of attempting to sunc mailboxes...).

I now use NetApp filers on which my maildirs live, and I have multiple IMAP/POP (and Horde/IMP webmail) servers living behind a pair of LVS directors. Using maildir format [0], I export the mailboxes via NFS to the frontend servers and it works a treat - generally because maildir saves files using filenames derived from the hostname, so the frontend servers don't hit race conditions when trying to manipulate files.

If you need really (and I mean _really_) high NFS transaction rates and fault-tolerance, I can't recommend the NetApp kit highly enough. If you don't, then other options - cheaper ones! - are (for example) high-end Intel-based IBM, Dell, HP/Compaq etc. servers with large hardware RAID arrays for redundancy. You'd have to address the various benchmarks according to your means, but there's a price point for every pocket somewhere.

[0] for the inevitable extra question: Exim v4.x for the (E)SMTP server, Courier IMAP server for IMAP and POP; all backended with MySQL for mailbox/address lookups and POP/IMAP/SMTP authentication.

Gabriel Neagoe Gabriel (dot) Neagoe (at) snt (dot) ro

for syncing the passwords - IF THE ENVIRONMENT IS SAFE- you could use NIS or rdist

You will not be explicitely configuring identd in an LVS. However authd/identd is used by sendmail and tcpwrappers and will cause problems. Services running on realservers can't use identd when running on an LVS (see identd and sendmail). Running identd as an LVS service doesn't fix this.

sendmail

in sendmail.cf set the value

Timeout.ident=0

Also see Why do connections to the smtp port take such a long time?

qmail qmail:

Martin Lichtin lichtin (at) bivio (dot) com

exim

Michael Stiller ms (at) 2scale (dot) net 26 Jun 2003

in exim.conf set the timeout for identd/auth to zero

rfc1413_query_timeout = 0s

Remember to reHUP exim.

Here's an LVS-DR with sendmail listening on the VIP on the realserver. Notice that the reply from the MTA includes the RIP allowing you to identify the realserver.

Connect from the client to VIP:smtp

client:~# telnet lvs.cluster.org smtp
 trying 192.168.1.110...
 Connected to lvs.cluster.org
 Escape character is '^]'.
220 lvs.cluster.org ESMTP Sendmail 8.9.1a/8.9.0; Sat 6 Nov 1999 13:16:30 GMT
 HELO client.cluster.org
250 client.cluster.org Hello [email protected] [192.168.1.12], pleased to meet you
 quit
221 client.cluster.org closing connection

check that you can access each realserver in turn (here the realserver with RIP=192.168.1.12 was accessed).

pop3 - as for smtp. The mail agents must see the same /home file system, so /home should be mounted on all realservers from a single file server.

Using qmail -

Abe Schwartz sloween (at) hotmail (dot) com

Has anyone used LVS to balance POP3 traffic in conjunction with qmail?

Wayne wayne (at) compute-aid (dot) com 13 Feb 2002

We used LVS to balance POP3 and Qmail without any problem.

Mike McLean mikem (at) redhat (dot) com 21 Apr 2004

Generally RW activity like with POP does not work well with LVS. When such things are required, one reasonable solution is to have a three tier setup where the lvs director load balances across multiple servers (in your case pop/imap), which access their data via a highly available shared filesystem/database. Of course this requires more machines and more hardware.

Kelly Corbin kcorbin (at) theiqgroup (dot) com 23 Aug 2004

Pop-Before-SMTP and LVS

I just added pop and smtp services to my load balancers, and I wanted to know if there was a way to tie the two connections together somehow. I use pop-before-smtp to allow my users to send mail, but sometimes they reconnect to a different server for smtp than the one they pop-ed into. Most of the time it's OK because they have pop-ed to both of the servers in the cluster but every now and then I have a user with an issue.

Is there a way to have it keep track of the IP and always send those SMTP and POP connections to the same realserver?

Josh Marshall josh (at) worldhosting (dot) org

You don't need to have the smtp and pop connections on the same realserver. We've got three mail servers here with pop-before-smtp running... I just point the daemons at the same file on an nfs share and it all just works. The pop-before-smtp daemons can handle all writing to the same file just fine.

Joe

You can link the two services with fwmark (and possibly persistence, depending on the time between connections). LVS is not really designed for writing to the realservers. To do so, you have to serialise the writes and then propagate the writes to the other realservers. This is a bit of a kludge, but is the way everyone handles writes to an LVS. The other point of an LVS is load balancing. What you're proposing is to turn off the loadbalancing and have the content on each of the realservers different. You need to be able to failout a realserver.

What you're proposing can be done, but it isn't an LVS anymore

Ramon Kagan rkagan (at) YorkU (dot) CA 20 Nov 2002

Previously we had been using DNS shuffle records to spread the load of IMAP connections across 3 machines. Having used LVS-DR for web traffic for a long time now (about 2.5 yeras) we decided to bring our IMAP service into the pool. The imap servers have been setup to retain idle connections for 24 hours (not my choice but the current setup anyways). The lvs servers have been setup to have a persistance of 8 hours. What has been happening lately is that users get the following type error.

"The server has disconnected, may have gone down or may be a network problem, try connecting again"

So, users don't have to reauthenticate per se, they just need to reclick on whatever button, so that the client they are using reauthenticates them. The timeout for this is as low as ten minutes. However the imap daemon is kept alive, and the client is shipped the PID of the imapd to reconnect directly. This is supposed to circumvent the necessity of reauthentication. In the case where the connection is lost, the reauthentication actually starts a new imapd process.

Joe

there is an LVS timeout and another timeout (tcp-keepalive) associated with tcp connections which affects telnet, ssh sessions. You need to reset both (see timeouts).

The solution: On the linux imap servers the tcp-keepalive value in /proc/sys/net/ipv4/tcp_keepalive_time is set to 7200

centaur# pwd /proc/sys/net/ipv4 centaur# cat tcp_keepalive_time 7200

This must be matched by the ipvsadm tcp timeout. So the options are to do one of:

• use ipvsadm --set 7200 0 0 on the lvs server

• echo "900" >! /proc/sys/net/ipv4/tcp_keepalive_time

Ramon Kagan rkagan (at) YorkU (dot) CA 27 Nov 2002

Further update on the solution: although this works there is one catch. If a person logs out and comes back within the 7200s as stated in the previous email, they continue to get the message because the realserver and the director don't match again. We will be lowering the value to somewhere between 5 and 15 minutes (300-900) to address this type of usage.

Torsten Rosenberger rosenberger (at) taoweb (dot) at 16 Sep 2003

i want to build a mail cluster with cyrus-imapd but i don't know how to handle the mailbox database on LVS.

Kjetil Torgrim Homme kjetilho (at) ifi (dot) uio (dot) no 16 Sep 2003

we use HP ServiceGuard for clustering (12 Cyrus instances running on three physical servers, storage on HP VA7410 connected through SAN), Perdition as a proxy (connects user to correct instance, also acts as an SSL accelerator, runs on two physical servers) and LVS (keepalived, active-active on two physical servers).

the other route is to use Cyrus Murder. e.g. shared folders will probably work better with Murder. administration is easier, too. we have to connect to correct instance. at the time we chose our architecture, Murder seemed unfinished, and we were worried about single point of failure. I'm afraid I haven't followed its development closely.

(another variation on the many reader/many writer problem)

loc (at) indochinanet (dot) com wrote:

I need this to convince my boss that LVS is the solution for very scalable and high available mail/POP server.

Rob Thomas rob (at) rpi (dot) net (dot) au

This is about the hardest clustering thing you'll ever do. Because of the constant read/write access's you -will- have problems with locking, and file corruption.. The 'best' way to do this is (IMHO):

1. NetCache Filer as the NFS disk server.
2. Several SMTP clients using NFS v3 to the NFS server.
3. Several POP/IMAP clients using NFS v3 to the NFS server.
4. At least one dedicated machine for sending mail out (smarthost)
5. LinuxDirector box in front of 2 and 3 firing requests off

Now, items 1 2 -and- 3 can be replaced by Linux boxes, but, NFS v3 is still in Alpha on linux. I -believe- that NetBSD (FreeBSD? One of them) has a fully functional NFS v3 implementation, so you can use that.

The reason why I emphasize NFSv3 is that it -finally- has 'real' locking support. You -must- have atomic locks to the file server, otherwise you -will- get corruption. And it's not something that'll happen occasionally. Picture this:

[client] -- [ l.d ] -- [external host] | [smtp server]-+-[pop3 server] | [filesrv]

Whilst [client] is reading mail (via [pop3 server]), [external host] sends an email to his mailbox. the pop3 client has a file handle on the mail spool, and suddenly data is appended to this. Now the problem is, the pop3 client has a copy of (what it thinks) is the mail spool in memory, and when the user deletes a file, the mail that's just been received will be deleted, because the pop3 client doesn't know about it.

This is actually rather a simplification, as just about every pop3 client understands this, and will let go of the file handle.. But, the same thing will happen if a message comes in -whilst the pop3d is deleting mail-.

POP Client SMTP Client I want to lock this file <-- I want to lock this file <-- You can lock the file --> You can lock the file --> Consider it locked <-- File is locked --> Consider it locked <-- Ooh, I can't lock it -->

The issue with NFS v1 and v2 is that whilst it has locking support, it's not atomic. NFS v3 can do this:

I want to lock this file <-- I want to lock this file <-- File is locked --> Ooh, I can't lock it -->

That's why you want NFSv3. Plus, it's faster, and it works over TCP, rather than UDP 8-)

This is about the hardest clustering thing you'll ever do.

Stefan Stefanov sstefanov (at) orbitel (dot) bg

I think this might be not-so-hardly achieved with CODA and Qmail.

Coda (http://www.coda.cs.cmu.edu) allows "clustering" of file system space. Qmail's (http://www.qmail.org) default mailbox format is Maildir, which is very lock safe format (even on NFS without lockd).

(I haven't implemented this, it's just a suggestion.)

Fabien fabien (at) oeilpouroeil (dot) org 03 Oct 2002

I successfully installed and tested a LVS-DR in a small network (1 director and 2 realservers) with http and pop3 load balancing/high availability, using too some hints from ultramonkey project (which team I thank too for the very good howto :)).

About the pop3 LVS here is what I did, and if someone feels like correcting me or suggesting/advicing me something better I will be grateful !

I used on both realservers : - the smtp daemon postfix with the VDA patch to handle maildir.

Here LVS doesn't handle smtp, I just use dns multiple MX feature. - the light weighted pop3 daemon tpop3d ( http://www.ex-parrot.com/~chris/tpop3d/ ) which can manage maildir.

The incoming mails are stored on the realservers in maildir type account following dns MX shedulding, and so are stored quite randomly on each realserver. To synchronize both realservers so that pop3 accounts are correct when checked, I use rsync and especially drsync.pl rsync wrapper ( http://hacks.dlux.hu/drsync/ ) which keeps track of a given filelist between rsync synchronizations (here the filelist is all the maildirs content).

At the moment, using crontab, drsync is run on each realserver every minutes and synchronizes the content of the other realserver with his one. It seems to work with a dozen pop3 accounts and hundreds of mail sent (no loss).

Fabien fabien (at) oeilpouroeil (dot) org 03 Oct 2002

I have an LVS-DR (1 director and 2 realservers) with http and pop3 load balancing/high availability (using some hints from ultramonkey project). On both realservers I have

• the smtp daemon postfix with the VDA patch to handle maildir. (LVS doesn't handle smtp, I just use the dns multiple MX feature.)
• the lightweight pop3 daemon tpop3d which can manage maildir format.

The incoming mails are stored on random realservers in maildir format following dns MX scheduling. To synchronize both realservers (so that pop3 accounts are correct when checked), I use rsync and especially the drsync.pl rsync wrapper, which keeps track of a given filelist between rsync synchronizations (here the filelist is the maildir files). drsync is run on each realserver every minute (cron) synchronizing the content with the other realserver(s). So far it works with a dozen pop3 accounts and hundreds of mail sent with no loss.

trietz trietz (at) t-ipnet (dot) net 18 Oct 2006

I'm using LVS-NAT for a simple rr-loadbalancing between two sendmail realservers. I setup a director with 3 NICs, one for the external connection(eth0) and the other two(eth1 and eth2) for connecting my realservers over crosspatch cable. The director has two ip adresses on the external interface. Here is the output from ipvsadm-save:

-A -t x.x.x.123:smtp -s rr
-a -t x.x.x.123:smtp -r 192.168.0.1:smtp -m -w 1
-a -t x.x.x.123:smtp -r 192.168.0.2:smtp -m -w 1
-A -t x.x.x.122:smtp -s rr
-a -t x.x.x.122:smtp -r 192.168.0.1:smtp -m -w 1
-A -t x.x.x.123:imaps -s rr
-a -t x.x.x.123:imaps -r 192.168.0.1:imaps -m -w 1

The packets intialized by the realservers are SNATed with iptables on the director successfully. My problem: loadbalancing works fine, but I see a lot of the reply packets from the realserver leaving the director on interface eth0 with the internal ips 192.168.0.1 and 192.168.0.2.

After testing I assume it is a timeout problem.

My solution:

Horms explanation (off-list)

Basically the real-server and end user have a connection open. It idles for a long time. So long that the connection entry on the linux director expires. Then a the real-server sends a packet over the connection, which the linux director doesn't recognise and sends out to the ether without unnatting it.

Solution? Well other than the one he suggests, changing the timeouts would help, assuming his analysis is correct.

Joe - this timeout problem came up earlier.

Dominik Klein dk (at) in-telegence (dot) net 15 May 2006

So let's just say we have a simple setup:

client -> switch -> director -> switch -> realserver

The client establishes a connection, sends some data, whatever. i Then it does nothing for say 10 minutes. After that it tries to reuse the still established connection and it works just fine. Then it does nothing for say 16 minutes and tries again to use the still established connection. In the meantime, the default timeout for the connection table (default 15 minutes) runs out and so this connection is not valid on the director. So the director replies with RST on the PSH packet from the client and the connection breaks for the client. The realserver does not know anything about the reset on the director, so it still considers the connection established.

The client opens a new connection, but the old one will still be considered established on the realserver. That's what made my MySQL server hit the max_connection limit and reject any new clients. I will try to set the timeout higher, as it can easily happen that my clients do nothing for a few hours at night, which will - sooner or later - hit the max_connection limit again.

Peter Mueller pmueller (at) sidestep (dot) com 10 May 2001

what open source mail programs have you guys used for SMTP mail farm with LVS? I'm thinking about Qmail or Sendmail?

Michael Brown Michael_E_Brown (at) Dell (dot) com, Joe and Greg Cope gjjc (at) rubberplant (dot) freeserve (dot) co (dot) uk 10 May 2001

You can do load balancing against multiple mail servers without LVS. Use multiple MX records to load balance, and mailing list management software (Mailman, maybe?). DNS responds with all MX records for a request. The MTA should then choose one at random from the same piority. (A cache DNS will also return all MX records.) You don't get persistent use of one MX record. If the chosen MX record points to a machine that's down, the MTA will choose another MX record.

Wensong

I think that central load balancing is more efficient in resource utilization than randomly picking up servers by clients, basic queuing theory can prove this. For example, if there are two mail servers grouped by multiple DNS MX records, it is quite possible that a mail server of load near to 1 still receiving new connections (QoS is bad here), in the mean while the other mail server just has load 0.1. If the central load balancing can keep the load of two server around 0.7 respectively, the resource utilization and QoS is better than that of the above case. :)

Michael Brown Michael_E_Brown (at) Dell (dot) com 15 May 2001

I agree, but... :-)

1. You can configure most mail programs to start refusing connections when load rises above a certain limit. The protocol itself has built-in redundancy and error-recovery. Connections will automatically fail-over to the secondary server when the primary refuses connections. Mail will _automatically_ spool on the sender's side if the server experiences temporary outage.
2. Mail service is a special case. The protocol/RFC itself specified application-level load balancing, no extra software required.
3. Central load balancer adds complexity/layers that can fail.

I maintain that mail serving (smtp only, pop/imap is another case entirely) is a special case that does not need the extra complexity of LVS. Basic Queuing theory aside, the protocol itself specifies load-balancing, failover, and error-recovery which has been proven with years of real-world use.

LVS is great for protocols that do not have the built-in protocol-level load-balancing and error recovery that SMTP inherently has (HTTP being a great example). All I am saying is use the right tool for the job.

Note this discussion applies to mail which is being forwarded by the MTA. The final target machine has the single-writer, many-reader problem as before (which is fine if it's a single node), (i.e. don't run the leaf node as an LVS).

Joe

How would someone like AOL handle the mail farm problem? How do users get to their mail? Does everyone in AOL get their mail off one machine (or replicated copies of it) or is each person directed to one of many smaller machines to get their mail?

Michael Brown

Tough question... AOL has a system of inbound mail relays to receive all their user's mail. Take a look:

[mebrown@blap opt]$ nslookup
Default Server:  ausdhcprr501.us.dell.com
Address:  143.166.227.254

> set type=mx
> aol.com
Server:  ausdhcprr501.us.dell.com
Address:  143.166.227.254

aol.com	preference = 15, mail exchanger = mailin-03.mx.aol.com
aol.com	preference = 15, mail exchanger = mailin-04.mx.aol.com
aol.com	preference = 15, mail exchanger = mailin-01.mx.aol.com
aol.com	preference = 15, mail exchanger = mailin-02.mx.aol.com
aol.com	nameserver = dns-01.ns.aol.com
aol.com	nameserver = dns-02.ns.aol.com
mailin-03.mx.aol.com	internet address = 152.163.224.88
mailin-03.mx.aol.com	internet address = 64.12.136.153
mailin-03.mx.aol.com	internet address = 205.188.156.186
mailin-04.mx.aol.com	internet address = 152.163.224.122
mailin-04.mx.aol.com	internet address = 205.188.158.25
mailin-04.mx.aol.com	internet address = 205.188.156.249
mailin-01.mx.aol.com	internet address = 152.163.224.26
mailin-01.mx.aol.com	internet address = 64.12.136.57
mailin-01.mx.aol.com	internet address = 205.188.156.122
mailin-01.mx.aol.com	internet address = 205.188.157.25
mailin-02.mx.aol.com	internet address = 64.12.136.89
mailin-02.mx.aol.com	internet address = 205.188.156.154
mailin-02.mx.aol.com	internet address = 64.12.136.121
dns-01.ns.aol.com	internet address = 152.163.159.232
dns-02.ns.aol.com	internet address = 205.188.157.232

So that is on the recieve side. On the actual user reading their mail side, things are much different. AOL doesn't use normal SMTP mail. They have their own proprietary system, which interfaces to the normal internet SMTP system through gateways. I don't know how AOL does their internal, proprietary stuff, but I would guess it would be massively distributed system.

Basically, you can break down your mail-farm problem into two, possibly three, areas.

1) Mail receipt (from the internet)
2) Users reading their mail
3) Mail sending (to the internet)

Items 1 and 3 can normally be hosted on the same set of machines, but it is important to realize that these are separate functions, and can be split up, if need be.

For item #1, the listing above showing what AOL does is probably a good example of how to set up a super-high-traffic mail gateway system. I normally prefer to add one more layer of protection on top of this: a super low-priority MX at an offsite location. (example: aol.com preference = 100, mail exchanger = disaster-recovery.offsite.aol.com )

For item #2, that is going to be a site policy, and can be handled many different ways depending on what mail software you use (imap, pop, etc). The good IMAP software has LDAP integration. This means you can separate groups of users onto separate IMAP servers. The mail client then can get the correct server from LDAP and contact it with standard protocols (IMAP/POP/etc).

For item #3, you will solve this differently depending on what software you have for #2. If the client software wants to send mail directly to a smart gateway, you are probably going to DNS round-robin between several hosts. If the client expects it's server (from #2) to handle sending email, then things will be handled differently.

Wenzhuo Zhang wenzhuo (at) zhmail (dot) com

Here's an article on paralleling mail servers by Derek Balling.

Shain Miley 25 May 2001

I am planning on setting up an LVS IMAP cluster. I read some posts that talk about file locking problems with NFS that might cause mailbox corruption. Do you think NFS will do the trick or is there a better (faster, journaling) file system out there that will work in a production environment.

Matthew S. Crocker matthew (at) crocker (dot) com 25 May 2001

NFS will do the trick but you will have locking problems if you use mbox format e-mail. You *must* use MailDir instead of mbox to avoid the locking issues.

You can also use GFS (www.globalfilesystem.org) which has a fault tolerant shared disk solution.

Don Hinshaw dwh (at) openrecording (dot) com

I do this. I use Qmail as it stores the email in Maildir format, which uses one file per message as opposed to mbox which keeps all messages in a single file. On a cluster this is an advantage since one server may have a file locked for writing while another is trying to write. Since they are locking two different files it eases the problems with NFS file locking.

Courier also supports Maildir format as I believe does Postfix.

I use Qmail+(many patches) for SMTP, Vpopmail for a single UID mail sandbox (shell accounts my ass, not on this rig), and Courier-Imap. Vpopmail is configured to store userinfo in MySQL and Courier-Imap auths out of Vpopmail's tables.

Joe:

I've always had the creeps about pop and imap sending clear text passwds. How do you handle passwds?

It's a non-issue on that particular system, which is a webmail server. There is no pop, just imapd and it's configured to allow connections only from localhost. The webmail is configured to connect to imapd on localhost. No outside connections allowed.

But, this is another reason that I started using Vpopmail. Since it is a mail sandbox that runs under a single UID, email users don't get a shell account, so even if their passwords are sniffed, it only gets the cracker a look into that user's mailbox, nothing more.

At least on our system. If a cracker grabs someone's passwd and then finds that the user uses the same passwd on every account they have, there's not much I can do about that.

On systems where users do have an ftp or shell login, I make sure that their login is not the same as their email login and I also gen random passwords for all such accounts, and disallow the users changing it.

I'm negotiating a commercial contract to host webmail for a company (that you would recognize if I weren't prohibited by NDA from disclosing the name), and if it goes through then I'll gen an SSL cert for that company and auth the webmail via SSL.

You can also support SSL for regular pop or imap clients such as Netscape Messanger or MS Outlook or Outlook Express.

Everything is installed in /var/qmail/* and that /var/qmail/ is an NFS v3 export from a RAID server. All servers connect to a dedicated MySQL server that also stores it's databases on another NFS share from the RAID. Also each server mounts /www from the RAID.

Each realserver runs all services, smtpd, imapd, httpd and dns. I use TWIG as a webmail imap client, which is configured to connect to imapd on localhost (each server does this). Incoming smtp, httpd and dns requests are load balanced, but not imapd, since they are local connections on each server. Each server stores it's logs locally, then they are combined with a cron script and moved to the raid.

It's been working very well in a devel environment for over a year (round- robin dns, not lvs). I've recently begun the project to rebuild the system and scale it up into a commercially viable system, which is quite a task since most of the software packages are at least a year old, and I'll be using a pair of LVS directors instead of the RRDNS.

Matthew Croker

Users will also be using some sort of webmail (IMP/HORDE) to get their mail when they are off site...other than that standard Eudora/Netscape will be used for retrieval.

I settled on TWIG mainly because of it's vhost support. With Vpopmail, I can execute

# /var/qmail/vpopmail/bin/vadddomain somenewdomain.com <postmaster passwd>

and add that domain to dns and begin adding users and serving it up. I had to tweak TWIG just a bit to get it to properly deal with the "user@domain" style login that Vpopmail requires, but otherwise it works great. Each vhost domain can have it's own config file, but there is only one copy of TWIG in /www. TWIG uses MySQL, and though it doesn't require it, I also create a seperate database for each vhost domain.

IMP's development runs along at about Mach 0.00000000004 and I got tired of waiting for them to get a working addressbook. That plus it doesn't vhost all that well. SquirrelMail is very nice, but again not much vhost support. Plus TWIG includes the kitchen sink. Email, contacts, schedule, notes, todo, bookmarks and even USENET (which I don't use), each module can be enabled/disabled in the config file for that domain, and it's got a very complete advanced security module (which I also don't use). It's all PHP and using mod_gzip is pretty fast. I tested the APC optimizer for PHP, but every time I made a change to a script I had to reload Apache. Not very handy for a devel system, but it did add some noticable speed increases, until I unloaded it.

(Joe - I've lost track of who is who here)

The realservers would need access to both the users home directories as well as the /var/mail directory. I am not too familiar with the actual locking problems...I understand the basics but I also hear that NFS V3 was supposed to fix some of the locking issues with V2...I also saw some links to GFS,AFS,etc not too sure how they would work...

For those of you that missed the importance of using Maildir format...

Alexandra Alvarado

I need to implement a cluster mail server with 2 computers smtp, 2 computers pop3 and 2 computers (NAS) failover for storage. The idea is to have duplicated copies of the mail (/var/spool/mail and /var/spool/mqueue) in the nas using online replication.

Matthew Crocker matthew (at) crocker (dot) com 20 May 2003

Do not use mbox mailbox format. mbox does not work well over NFS with multiple hosts writing to the same file. You should use Maildir format. Just about every SMTP server can handle Maildir.

I'm running 4 mail servers. All mail servers have SMTP,POP3,IMAP, SMTPS,IMAP-SSL,POP3-SSL running on them. I'm using qmail, qmail-ldap, and Courier-IMAP. I have a Network Appliance NetFiler F720 200gig NFS server for my Maildir storage I have 3 OpenLDAP servers setup with 1 master and 2 slaves. The Mail servers only talk with the slaves. All mailAddress information is held in the LDAP directory.

You need to centralize the storage of mail using NAS and centralized the storage of account information using either LDAP or MySQL

How do you plan on failing over the NAS? NFS with soft mounts should handle it pretty well. Setup 2 Linux boxes one as an active, one as a passive NFS server. Both connected via Fiber Channel to a chunk of drives. The passive machine *must* have a way to STOMITH (Shoot The Other Machine In The Head) the active server if it crashes. You do not want to have both machines mounting the same drive space at the same time. Very bad things will happen....

Soo.. Setup an EXT3 filesystem on the FC drives. mount it on the active Linux box, export it over NFS with a virtual IP address. If the active server fails you need to remove power from it using a remote power switch. You need to be able to guarantee that it won't come back to life and start writing to the filesystem again. Clean the FC filesystem. remount it and export it over NFS on the same virtual IP addresses. keepalived can handle the VIP stuff with VRRP. I think it can also launch an external script during a failover to handle the shooting, cleaning, mounting, exporting of the filesystem. EXT3 cleans pretty quickly.

The SMTP/POP3 servers will be very un-happy to see their NFS server disappear so you will need to recover quickly. Processes will probably pile up in 'D'isk wait status on all of the machines. load will go through the roof. After the NFS server comes back online the hung processes should recover and finish up.

Adaptec makes a very nice 12 disk rack mounted RAID controller that has U160 SCSI going to the disks and 1 gig FiberChannel going to the servers. Redundant power, Redundant RAID controllers, Redundant FC loops going to each server. It is called the DuraStor 7320S. Plan on about $10kUS + drives for this type of system.

Network Appliance make an amazing box with complete High Availability failover of clustered data. You can expect to pay $200KUS for a complete clustered solution with 300GB usable storage. Fully redundant. Pretty much shoot a shotgun at it and not go down or lose data.

EXT3 running on Logical Volume Manager (LVM) can handle journalling and snapshots

Making the servers/services redundant is easy. Making your NAS/SAN redundant is expensive.

I'm looking into the Adaptec external RAID controller/drive array setup with 2 Linux boxes for my NAS. I've been running my NetApp for 3 years and have *NEVER* had it crash. It really is an amazing box. The problem is it is only one box and I don't have $200k to make it a cluster. I think I can do a pretty good job for about $20k with the Adaptec box, a bunch of Seagate drives and a couple Linux boxes.

You could also look into distributed filesystems like GFC, Coda ... but I don't feel confident enough in them to handle production data just yet.

Just a quick note: About a week ago I tried compiling a kernel that had been patched by SGI for XFS. The kernel (2.4.2) compiled fine, but choked once the LVS patches had been applied. Not having a lot of time to play around with it, I simply moved to 2.4.4+lvs 0.9 and decided not to bother with XFS on the director boxes.

also I thought about samba and only found one post from last year where someone was going to try it but there was no more info there.

Well, there's how I do it. I've tried damned near every combination of GPL software available over about the last 2 years to finally arrive at my current setup. Now if I could just load balance MySQL...

Greg Cope

MySQL connections / data transfere work much faster (20% ish) when on local host - so how about running mysql on each host, which is a select only system, and each localhost uses replication to a mster DB that is used for inserts and updates ?

Ultimately I think I'll have to. After I get done rebuilding the system to use kernel 2.4 and LVS and everything is stabilized, then I'll be looking very hard at just this sort of thing.

Joe, 04 Jun 2001

SMTP servers need access to DNS for reverse name lookup. If they are LVS'ed in a LVS-DR setup, won't this be a problem?

Matthew S. Crocker matthew (at) crocker (dot) com

You only need to make sure you have the proper forward and reverse lookup set. inbound mail to an SMTP server gets load balanced by the LVS but it still sees the orginal from IP of the sender and can do reverse lookups as normal. outbound mail from an SMTP server makes connections from its real IP address which can be NAT'd by a firewall or not. That IP address can also be reverse looked up.

normally the realservers in a LVS-DR setup have private IPs for the RIPs and hence they can't receive replies from calls made to external name servers.

I would also assume that people would write filter rules to only allow packets in and out of the realservers that belong to the services listed in the director's ipvsadm tables.

I take it that your LVS'ed SMTP servers can access external DNS servers, either by NAT through the director, or in the case of LVS-DR by having public IPs and making calls from those IPs to external nameservers via the default gw of the realservers?

We currently have our realservers with public IP addresses.

Bowie Bailey Bowie_Bailey (at) buc (dot) com

You can also do this by NAT through a firewall or router. I am not doing SMTP, but my entire LVS setup (VIPs and all) is private. I give the VIPs a static conduit through the firewall for external access. The realservers can access the internet via NAT, the same as any computer on the network.

Adail Oliveira

I want to build e-mail cluster using lvs, anybody have experience with this?

kwijibo (at) zianet (dot) com

It is pretty much the same as setting up an LVS for any other service. The biggest problem you will probably have is figuring out how handle storage for the mailboxes. My experience is that it works great. I am not sure how I would handle our mail load without it.

Todd Lyons tlyons (at) ivenue (dot) com 2005/27/05

Agreed. We use a NetApp for our central NFS server, 2 http machines for webmail, 2 imap machines, and 2 smtp machines. We have a 2 node load balancer with failover that balances the 3 protocols listed above (as well as other webservers). The 6 machines serving mail are dual P4 2.8 GHz with 1Gig RAM boxen, the load balancers are old P3 700 boxes that only do load balancing. We're just a small system though compared to many who do it. We estimate we could scale up to about 10-20 real servers for each service before we start to get throughput problems with the NetApp.

Graeme Fowler graeme (at) graemef (dot) net 15 Dec 2005

My day job sees me working for an ISP; we have a number of mail systems where we use multiple frontend servers (some behind LVS, some using other methods) with NetApp Filer backends offering mail storage over NFS mounts to the real/frontend servers. They handle SMTP, POP, IMAP, Webmail (and other things not necessarily mail related) but store the data on common NFS mounts.

Yes, there are well-documented issues with "mail on NFS" but that usually happens with shared SMTP server spools rather than IMAP/POP systems. We haven't had a problem yet; the filers are very reliable (and good if they do go wrong on rare occasions, too) and the platform scales out nicely.

Pierre Ancelot pierre (at) bostoncybertech (dot) com 15 Dec 2005

I tried to implement a load balance mail system with imap and imaps In this case, a user creating an imap folder will create it on only one node.... How do I update a mail received on one host to every other host? Using rsync would delete every mail received in the same time on other servers...

Scott J. Henson scotth () csee ! wvu ! edu 2005-12-15

Perdition or Courier-Imap. Both of them are/have imap proxies(or directors). So what you end up with is one(or more) front end proxies that send the person to the right machine. It looks something like this. Btw, we are moving to such a setup so I don't know how well it will be

The connection hits your load balancers(just straight ip_vs). Then it hands of the connection to a pool of imap proxies. Then each imap proxy figures out which real mail warehouse to send the message to(ldap is a good place to store this info cause it too can be load balanced, slaves anyway and then no need to load balance the master).

At that point the user does their thing and its all stored on one server. But you have many mail boxes distributed across many mail warehouses. There is an issue of backups and such for the mail warehouse, but this distributes the load so you can serve many more mail boxes than one server could. I think we are gonna go with RAID arrays and then hot spare mail warehouse mirrors. If the lead warehouse fails the backup comes online(through something like heartbeat). This should be more than enough redundancy for us, but you may want to look into other solutions if you want more, aka fiberchannel or some such.

Obviously this setup can get very complex but become very stable. Depending on your application you can throw money at it or not even have hot spares and trust in your RAID/backups.

Markmsalists () gmx ! net 2005-12-15

Maybe I am misinterpreting this, but it sounds like each mailbox is assigned to exactly one realserver? So you have distributed the mailboxes for load-balancing, but is there any redundancy if one of the boxes goes down?

Scott J. Henson scotth (at) csee (dot) wvu (dot) edu 15 Dec 2005

Nope, you're not misinterpreting anything. With mail you can't really have the type of redundancy you can have with say a webserver serving out content. The problem is that imap doesn't like to be distributed. The only way is to do it over something like iSCSI or fiberchannel or some other enterprise level storage medium. What we are doing is to distribute load and to provide some redundancy. If one mail box goes down then we bring up the hot spare, but most of our mail boxes still stay up. Also if one has a failed RAID, then we can move all the mail boxes off of it and bring it down, repair the raid, then bring it back up and move the mail boxes back. It really offers more flexibility and does increase the redundancy on a site level.

Todd Lyons tlyons () ivenue ! com 2005-12-16

Could always go with something like cyrus-imap with the murder extension, which is for an imap cluster. I've never set it up, don't know any details much beyond what I've stated here. But it's supposed to make the mail machines look like a cluster.

Kees Bos k.bos () zx ! nl 2005-12-16

Maybe you can put some kind of imapproxy in front of your imap servers. The imapproxy than has to know about the multiple imap servers and the imapproxy itself can be loadbalanced. I haven't used it myself, but perdition seems to do this: http://www.vergenet.net/linux/perdition/

Scott J. Henson scotth () csee ! wvu ! edu 2005-12-16

Yes, I forgot to mention that, but word from the already been there, its HARD. At least in my experience its more trouble than its worth.

Here's an example of a commercial mail farm using LVS. Commercial ventures are usually loathe to tell us how they use LVS - Seems they don't understand the spririt of GPL. Even after you're told the setup, you still need someone to get it going and keep it going, so I don't know what advantage they get out of keeping their setups secret. Michael Spiegle popped up on the mailing list and gave some info about a LVS'ed mail farm he'd setup for a customer, so I asked off-list if he'd mind giving us more details. So here it is. Thanks Michael.

Michael Spiegle mike (at) nauticaltech (dot) com 13 Nov 2006

The setup is VERY simple and straightforward. We have 17 mailservers in production right now. Here's some specs on our setup:

Pair of LVS-NAT directors, each dual Xeon, 2GB RAM, dual 3com Fiber 1000SX NICS, failover uses keepalived. There is no firewall in-front of the directors. The 17 mail servers are dual Xeon, 4GB RAM, dual 10K SCSI disks in RAID1 (software), dual onboard e1000.

During peak time today (monday has highest load), we were doing about 14K active connections at any given time (probably over 30K inactive connections). Actual bandwidth isn't terribly high... due to the nature of the service. The director pair also provides load balancing for a cluster of 30 Sun X1 servers running HTTPd, which has much fewer connections, but a little more bandwidth intensive. I think we might be pushing 350mbit/sec at peak times (haven't seen MRTG graphs in a while).

Our services allow local mailboxes and forwarding to external addresses. We have something in the neighborhood of 250K local mailboxes and 170K external forwards. The load on the director pair is practically non-existent (generally 0.0x).

The ONLY time we have EVER come close to maxing out our LVS was during a DDOS attack. The NICs we use don't have interrupt coalescence in the driver, so we were actually running out of interrupts on the box which isn't the fault of LVS. I don't recall any other metrics from the DDOS attack, but LVS would have swallowed it if we had interrupt coalescence.

The mailservers however have some issues which need to be sorted out. They run anywhere from 2 to 15 as the load average. They use their local disks heavilly for temporary incoming mail storage. We store all mail on a pair of NetApp FAS-940s which we appear to be pushing the limits on somehow ( the excessive amount of NFS-wait we're hitting is driving up the load on the mailservers).

How does localmail get from the realservers to the NetApp? The mailserver uses multiple daemons to accomplish mail tasks. When a piece of mail comes in, a message is created on the disk. Another daemon is dedicated to figuring out where the messages go (to an external forward, or to a local mailbox). If the message is going to an external forward, it is handled by an external-mail daemon. If the message is staying local, an internal-mail daemon puts it on the NetApp.

When a customer connects via POP, the POP server looks up the location of the mailbox in memory, then goes straight to the NetApp to fetch the messages. All NetApps are mounted to the realservers via Linux NFS client. (the excessive amount of NFS-wait we're hitting is driving up the load on the mailservers).

nfs used to be a real dog. It's impressive that nowadays you can run a network disk for a machine that's being pushed hard.

On any given server, we've got 300 items in dmesg regarding "couldn't contact NetApp". We have some issues to sort out regarding the NFS mount options we're using. Also, the NetApps have an issue where if a single mailbox hits the dirsize limits (I think its something on the order of 2 million messages in a single directory), the NetApp freaks out and pegs 100% CPU. During this time, load on the mailservers doubles because those mail processes/daemons can't talk to the NetApp and causes a pile-up of connections.

It is a real "cluster" in the sense that customer data (email address to local mailbox mapping) is stored in a memory-resident database - therefore, access is VERY fast and any server can handle any number of connections.

We have a pair of dual-xeon boxes running tinyDNS to provide caching DNS for the mail cluster. Previously, we had our default gateways on the mailservers set to the 2nd interface of the directors (which were also load balanced). This caused all cache DNS traffic to go through the LVS which resulted in a conntrack table of over 100K at peaktimes. Even though the DNS traffic was UDP, netfilter attempts to create a very basic connection status for the traffic. For example, if mailserver01 sends out a DNS request to dnscache01, netfilter will create a conntrack table entry and label it as unack'd until LVS sees a response from dnscache01. When it sees this response, it relabels the connection to ack'd. Once I realized what was going on, I re-architected the DNS layout slightly to allow the mailservers to directly communicate with the caching nameservers. Now, the LVS runs about 20-25K connections in conntrack.

About the limitations of conntrack. Previously (year ago), we had an old pair of DNS caches, which were nothing more than single-proc P3 boxes @ 900Mhz. We NEVER had a problem with these boxes until we provisioned the new cache boxes (dual xeon). The old boxes were on the same VLAN as the mailservers, so traffic/connections through LVS weren't really a problem. When we provisioned the new DNS caches, which were to be "segregated" from other internal networks, thus the mailservers using LVS as a gateway to hit them. I always had a feeling that our mail servers were "slower" with the new caches than the old ones. I dug deeper one day and found out that we were hitting conntrack limits on LVS from all these DNS queries. I alleviated that as I explained earlier by re-architecting the DNS cache layout, but I still don't feel it was quite up to snuff. I did notice that each of the new DNS caches had netfilter enabled in kernel (unnecessary addition from our shoddy development team) and we were hitting conntrack limits on them as well.

The moral of the story is that I never had these problems with the old servers, because I compiled the kernel WITHOUT netfilter. So in the old kernel

Its not very scientific, but I believe conntrack introduces enough latency to be noticeable at our level.

Since we run a "real" cluster, we can fail out any machine as we please. Any mailserver can handle any customer.

We also have a separate LVS-DR web cluster based on linux x86. It runs the same memory-resident database to build apache virtualhost entries on the fly. Also, since each customer has their own IP (for SSL purposes), the realservers have about 65K (250+ class Cs) bound to the loopback. Works beautifully.

How do you handle failover of large numbers of VIPs?

Since I haven't had a chance to work on that particular system yet, I don't know. I can tell you however that the slave wouldn't have to ARP for all of those IPs in our particular setup. We have a pair of PIX firewalls in front of the LVS which do passthru to the LVS. The only thing the slave has to ARP for is the placeholder IP on the interface, and the firewall will figure out where to send the traffic. True, if the master firewall died, the slave would have to ARP for all the IPs... but we've found the firewalls to be quite reliable.

I'm hoping to be able to push LVS a little farther with a possible project in the upcomming month. I'm leaving this place I currently work at and am going to a company that does lots of media streaming. They're using netscalers to push 10gbit of bandwidth outbound in an asymmetrical-routing sort of way. Since they cost 80K/ea, I'm hoping I can convince them that LVS is just as good if not better for MUCH cheaper.

Noah Roberts wrote:

When I use urls like www.myserver.org/directory/ everything works fine. But if I don't have the ending / then my client attempts to find the realserver and ask it, and it uses the hostname that I have in /etc/hosts on the director which is to the internal LAN so it fails badly.

Scott Laird laird (at) internap (dot) com 02 Jul 2001

Assuming that you're using Apache, set the ServerName for the realserver to the virtual server name. When a user does a 'GET /some/directory', Apache returns a redirect to 'http://$servername/some/directory/'.

nigel (at) turbo10 (dot) com

This weekend the web service we run came under increased load --- about an extra 10,000,000 queries per day ---- when InActConn went from 200-300 to 2000+ in about 60 seconds and the LVS locks up.

Rob ipvsuser (at) itsbeen (dot) sent (dot) com 2005/03/13

I had a high number of inactive connections with apache set up to not use keepalive at all. After activating keep alive in apache (LVS was already persistant) the number of inactive connections went way down.

So in my case at least, connections were setup, used for a single GET for a gif, button, jpeg, js script, or other page component then the server closed the connection, only to open another for the next gif, etc.

You might be able to use something like multilog to watch a bunch of the logs at the same time to get an idea if the traffic looks like real people (get page 1, get page 1 images, get page 2, get page 2 images) or if it is random hammering from a dos attack.

I wrote a small shell script that pulled the recent log entries, counted the hits per IP address for certain requests and then created a iptables rule on the director (or some machine in front of the director) to tarpit requests from that IP. This worked in my situation because we knew that certain URLs were only hit a small number of times during a legit use session (like a login page shouldn't be hit 957 times in an hour by the same external IP) This could help reduce the tide of requests if you are actually encountering a (d)dos. I ran it every 12 minutes or so. If you are getting ddos'd the tarpit function of iptables http://www.securityfocus.com/infocus/1723 or the tarpit standalone can be a great help. Also, Felix and his company seem to have helped some large companies deal with high traffic ddos attacks - http://www.fefe.de/

BTW, You might be interested in http://www.backhand.org/mod_log_spread/ for centralized and redundant logging. That way you can run different kinds of real time analysis with no extra load on the webservers or the normal logging hosts by just having an additional machine join/subscribe to the multicast spread group with the log data.

OK I can't find my script, but this was the start of it, it is hardly a shell script (but someone may find it useful): Add a "grep blah" command just before the awk '{print $2}' if you want just certain requests or other filtering.

multidaychk.sh
#!/bin/sh
# look for mutliday patterns
# $1 is how many days back to search
# $2 is how many high usage IPs to list
ls -1tr /usr/local/apache2/logs/access_log.200*0 | tail -${1} | xargs -n 
1 cat | awk '{print $2}' | sort | uniq -c | sort -nr | head -${2}

byhrchk.sh
#!/bin/sh
# looks for IPs hitting during a certain hr of the day
# $1 is how many days back to search
# $2 is how many high usage IPs to list
# $3 is which hour of the day
ls -1tr /usr/local/apache2/logs/access_log.200*0 | tail -${1} | xargs -n 
1 cat | fgrep "2005:${3}" | awk '{print $2}' | sort | uniq -c | sort -nr 
| head -${2}

recentchk.sh
#!/bin/sh
# This just checks the latest X lines from the newest log file
# $1 is how many lines from the file
# $2 is how many high usage IPs to list
ls -1tr /usr/local/apache2/logs/access_log.200*0 | tail -1 | xargs -n 1 
tail -${1} | awk '{print $2}' | sort | uniq -c | sort -nr | head -${2}

When first setting up, to check that your LVS is working...

You need to shut down httpd gracefully, by bringing the weight to 0 and letting connections drop, or you will not be able to bind to port 80 when you restart httpd. If you want to do on the fly modifications to your httpd, and keep all realservers in the same state, you may have problems.

Thornton Prime thornton (at) jalan (dot) com 05 Jan 2001

I have been having some problems restarting apache on servers that are using LVS-NAT and was hoping someone had some insight or a workaround.

Basically, when I make a configuration change to my webservers and I try to restart them (either with a complete shutdown or even just a graceful restart), Apache tries to close all the current connections and re-bind to the port. The problem is that invariably it takes several minutes for all the current connections to clear even if I kill apache, and the server won't start as long as any socket is open on port 80, even if it is in a 'CLOSING' state.

Michael E Brown wrote:

Catch-22. I think the proper way to do something like this is to take the affected server out of the LVS table _before_ making any configuration changes to the machine. Wait until all connections are closed, then make your change and restart apache. You should run into less problems this way. After the server has restarted, then add it back into the pool.

I thought of that, but unfortunately I need to make sure that the servers in the cluster remain in a near identical state, so the reconfiguration time should be minimal.

Julian wrote

Hm, I don't have such problems with Apache. I use the default configuration-time settings, may be with higher process limit only. Are you sure you use the latest 2.2 kernels in the realservers?

I'm guessing that my problem is that I am using LVS persistent connections, and combined with apache's lingering close this makes it difficult for apache to know the difference between a slow connection and a dead connection when it tries to close down, so the time it takes to clear some of the sockets approaches my LVS persistence time.

I haven't tried turning off persistence, and I haven't tried re-compiling apache without lingering-close. This is a production cluster with rather heavy traffic and I don't have a test cluster to play with. In the end rebooting the machine has been faster than waiting for the ports to clear so I can restart apache, but this seems really dumb, and doesn't work well because then my cluster machines have different configuration states.

One reason for your servers to block is a very low value for the client number. You can build apache in this way:

CFLAGS=-DHARD_SERVER_LIMIT=2048 ./configure ...

and then to increase MaxClients (up to the above limit). Try with different values. And don't play too much with the MinSpareServers and MaxSpareServers. Values near the default are preferred. Is your kernel compiled with higher value for the number of processes:

/usr/src/linux/include/linux/tasks.h

Is there any way anyone knows of to kill the sockets on the webserver other than simply wait for them to clear out or rebooting the machine? (I tried also taking the interface down and bringing it up again ... that didn't work either.)

Is there any way to 'reset' the MASQ table on the LVS machine to force a reset?

No way! The masq follows the TCP protocol and it is transparent to the both ends. The expiration timeouts in the LVS/MASQ box are high enough to allow the connection termination to complete. Do you remove the realservers from the LVS configuration before stopping the apaches? This can block the traffic and can delay the shutdown. It seems the fastest way to restart the apache is apachectl graceful, of course, if you don't change anything in apachectl (in the httpd args).

(From Ted I think)

Setup -

realservers are node1.foobar.com, node2.foobar.com... nodeN.foobar.com, director has VIP=lvs.foobar.com (all realservers appear as lvs.foobar.com to users).

Problem -

if you run the indexing program on one of the (identical) realservers, the urls of the indexed files will be

http://nodeX.foobar.com/filename

These urls will be unuseable by clients out in internetland since the realservers are not individually accessable by clients.

If instead you run the indexing program from outside the LVS (as a user), you will get the correct urls for the files, but you will have to move/copy your index back to the realservers.

Solution (from Ted Pavlic, edited by Joe).

On the indexing node, if you are using LVS-NAT add a non-arping device (eg lo:0, tunl0, ppp0, slip0 or dummy) with IP=VIP as if you were setting up LVS-DR (or LVS-Tun). With LVS-DR/VS-Tun this device with the VIP is already setup. The VIP is associated in dns with the name lvs.foobar.com. To index, on the indexing node, start indexing from http://lvs.foobar.com and the realserver will index itself giving the URLs appropriate for the user in the index.

Alternately (for LVS-NAT), on the indexing node, add the following line to /etc/hosts.

127.0.0.1       localhost lvs.foobar.com

make sure your resolver looks to /etc/hosts before it looks to dns and then run your indexing program. This is a less general solution, since if the name of lvs.foobar.com was changed to lvs.bazbar.com, or if lvs.foobar.com is changed to be a CNAME, then you would have to edit all your hosts files. The solution with the VIP on every machine would be handled by dns.

There is no need to fool with anything unless you are running LVS-NAT.

Noah Roberts wrote:

If anyone has had success with htpasswords in an LVS cluster please tell me how you did it.

Thornton Prime thornton (at) jalan (dot) com Fri, 06 Jul 2001

We have HTTP authentication working on dozens of sites through LVS with all sorts of different password storage from old fashioned htpasswd files to LDAP. LVS when working properly is pretty transparent to HTTP between the client and server.

Tony Requist

We currently have a LVS configuration with 2 directors and a set of web servers using LVS-DR and keepalived between the directors (and a set of MySql servers also). This is all working well using the standard RR scheduling without persistence. We will be adding functionality that will be storing data on some but not all web servers. For this, we need to be able to route requests to specific web servers according to the URL. Ideally I would generate URLs like:

stuff.domain.com://KEY

and I could have a little code in LVS (or called from LVS) where I could decode KEY to find that the data is on server A, B and C -- then have LVS route to one of these three servers. I've looked through the HOWTO and searched around but I have not been able to find anything.

Scott J. Henson scotth (at) csee (dot) wvu (dot) edu 20 Jul 2005

I would personally use apache proxy statements on the servers that don't have the information. This will increase load slightly, but is probably the easiest.

If you really want to go the LVS route, there are some issues, I believe. If my memory serves, the current version of LVS is a level 4 router and to do what you want, you would need a level 7 router. I have heard of some patches floating around to turn ip_vs into a level 7 router, but Ive not seen them personally, nor tried them.

	Note
	L7 L7 Switch requires much more computation than L4. You don't want to do anything at L7 that you can handle any way at all at L4.

Todd Lyons tlyons (at) ivenue (dot) com 20 Jul 2005

This is application level, not network level. The better solution (IMHO) is to put two machines doing reverse proxy with the rules to send the requests to the correct server. Then have your load balancers balance among the two rproxies.

A poor man's solution would be to put the reverse proxying on the webservers themselves.

This is not really good for HA though since you don't have redundancy if there is only one webserver serving a particular resource.

If the reverse proxies have a different IP than the public IP of the webservers, then you have more options.

Andres Canada

When the cluster node gets the request it looks to apache configuration and whatever to serve this request. When a Director node receives a request for a special web application that is only in one cluster node (for example node35) there should be something inside it that send that request to a "special node" (not the next one if it's using round robin, but the node35).

Todd Lyons tlyons (at) ivenue (dot) com Dec 14 2005

You should consider setting up a reverse proxy. This is a machine that sits in front of your apache boxen that examines urls and sends them to various private apache servers, and sends the reply back. The outside world doesn't talk directly to the private apache servers.

In our case, we have several different machines that handle different types of requests. We have 3 rp's sitting in front of them getting load balanced by two LVS boxen. The load balanced rp's receive the GET or POST from the outside world, examine it, and send the request to the appropriate private machine, waits for the reply, and sends that to the requesting client.

From Lars lmb (at) suse (dot) de mod_backhand, a method of balancing apache httpd servers that looks like ICP for web caches.

Jessica Yang, 8 Oct 2004

Our application require L7 load balancing because we use URL rewriting to keep the session info in the requested URLs, like this: http://ip/servlet/servletname;jsessionid =*****. Basically, we want the load balancer will deliver the requests who have the same jsessionid to the same realserver. Looking through the LVS document, KTCPVS seems to be able to provide L7 load balancing, but I couldn't find any documentation about compiling, configuring, features and commands of KTCPVS. Does KTCPVS have the feature to distinguish the jsessionid in the requested URL and/or in the Cookie header? Does KTCPVS have to be bundled together with IPVS? And what is the process to make it work?

Wensong 09 Oct 2004

KTCPVS has a cookie-injection load balancing feature just as you described. You can use something like the following

insmod ktcpvs.o
insmod ktcpvs_chttp.o
tcpvsadm -A -i http -s chttp
tcpvsadm -a -i http -r realserver1:80
tcpvsadm -a -i http -r realserver2:80
tcpvsadm -a -i http -r realserver3:80

Jacob Coby jcoby (at) listingbook (dot) com 08 Oct 2004

It almost sounds like you need to use a proxy instead of LVS to do the load balancing. If something in your jsessionid is unique to a server, it would be very simple to make a rewrite rule accomplish what you want.

Clint Byrum cbyrum (at) spamaps (dot) org 08 Oct 2004

mod_backhand (http://www.backhand.org/mod_backhand/).

VERY nice load balancing proxy module for apache. It does require that your content be served from Apache 1.3 (Windows or Unix) though.

cheaney Chen

There are a lot of different kinds of SLB techniques, ex. DNS-based, Dispatcher-based(like LVS), and server-based , etc. And my question is, for a commercial web site (like yahoo or ...) how to do SLB. What methods are used to handle huge numbers of client's requests? Combination of SLB techniques above , or ... ?!

Clint Byrum cbyrum (at) spamaps (dot) org 06 Jan 2005

I've used LVS for frontend balancing, and backhand (www.backhand.org) for backend.

In short, mod_backhand takes specific resource-intensive requests and proxies them to whichever servers are least busy. It works *VERY* well. We have a farm of cheap boxes serving lots of CPU intensive requests and every box has the same exact load average within 2-3%. It even allows persistence. Only downside is it requires Apache 1.3, but so far that hasn't been a problem for us. :)

With squids you can't use a VIP to set up a virtual service - the requests you're interested in are all going to a port, port 80. Since the requests are being to an IP that's not on the director, you also need to force the director to accept the packets for local processing (see routing to a director without a VIP). Both of these problems were handled in one blow in the 2.0 and 2.2 kernels by Horms method using the -j REDIRECT option to ipchains. This doesn't work for the 2.4 and beyond kernels, as the dst_addr of the packet is rewritten before the packet is delivered to the director. A possible (untested) solution is the TPROXY method.

The method used starting with the 2.4 kernels is to mark all packets to port 80 and schedule on the mark. The packets are accepted locally by iproute2 commands.

Con Tassios ct (at) swin (dot) edu (dot) au 13 Feb 2005

Transparent proxy with squid works well if you use fwmarks. I use it with the following LVS-DR configuration:

bikrant (at) wlink (dot) com (dot) np Jun 24 2005

   <cisco router>
    202.79.xx.230
       |
       |-------------------------|-----------------------|
       |                         |                       |
       |                         |                       |
eth0:202.79.xx.240/24 fxp0: 202.79.xx.241/24     202.79.xx.235/24
    <Director>           <realserver >             <client>
    (gw: cisco)           (gw: cisco)              (gw: cisco)

The default route for all machines is the router. Forwarding is by LVS-DR.

Director

Gentoo Linux with 2.6.10 Kernel

ipvsadm -A -f 1 -s sh
ipvsadm -a -f 1 -r 202.79.xx.241:80

(-g is the default, the :80 is ignored for LVS-DR)

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1

ip rule add prio 100 fwmark 1 table 100
ip route add local 0/0 dev lo table 100

echo 0 >  /proc/sys/net/ipv4/ip_forward

Real server Configuration: FreeBSD 5.3 with squid configured by trans-proxy.

Cisco Router:

interface Ethernet0/0
ip address 202.79.xx.230 255.255.255.0
ip policy route-map proxy-redirect

access-list 110
     access-list 110 deny tcp host 202.79.xx.241 any eq 80
     access-list 110 permit tcp 202.79.xx.0 0.0.0.255 any eq 80

route-map proxy-redirect permit 10
    match ip address 110
    set ip next-hop 202.79.xx.240

peterbaitz - 27 Jan 2003

Is it possible for SSL to be supported on the director rather than on the realserver(s)?

Right now, the powers that be have brought up the question of placing a purchased Mirapoint email system behind a "free" load balancer (neglecting to consider that Mirapoint runs on FreeBSD, and that Piranha is a purchasable LVS product as well).

Joe

I think you're saying that you want the director to forward port 80 and to accept port 443 locally, ie to not forward port 443. If this is the case then you add entries with ipvsadm for port 80 only. All other traffic sent to the VIP on other ports will be handled locally.

Matthew Crocker matthew (at) crocker (dot) com 27 Jan 2003

It can be done, in fact just about anything can be done these days. If it is a smart thing to do is another matter... What you are trying to do isn't really a function of LVS. You can setup Apache+SSL running in a reverse proxy configuration. That apache and be running on or in front of the LVS director. The apache can then make normal web connections to the internal machines which can be run through the LVS director and load balanced.

You can use keepalived or hearbeat to manage the high availability functions of your Apache/SSL proxy. You can use hardware based SSL engines to handle the encryption/decryption. This is all transparent to the functions of LVS.

LVS is 'just' a smart IP packet router, you give it a packet and tell it how you want it handled. It can be configured to do a bunch of things.

The ideal solution for the highest performance and greatest availability is to have 2 groups of directors, each group having N+1 machines running LVS. Have 1 group of Apache/SSL servers configured and 1 group of internal web servers.

LVS group 1 load balances the inbound SSL traffic to one of the Apache/SSL servers. The apache servers make connections to the internal servers though LVS group 2. LVS group 2 load balances the internel HTTP traffic into the realservers.

To save money you could move Apache/SSL onto the LVS directors but that could hurt performance.

Matt Venn lvs (at) attvenn (dot) net Jul 6 2004

You might want to do this if you have highly specced director(s) that you don't want to waste, or not much SSL traffic. I use this setup to cache all images, and to do SSL acceleration for my realservers. Requirements

Method:

patch ip_vs_core.c with Carlos' patch, configure the kernel for LVS, build kernel, install and reboot, compile and install ipvsadm-1.21.

Here are my config files for a small cluster with 1 director and 2 realservers. This config will do the SSL for traffic to editcluster.localnet, and load balance both https and http traffic to the 2 realservers.

#/etc/hosts
127.0.0.1               localhost
192.168.0.50            director1.localnet editcluster.localnet vhost1.localnet
192.168.1.1             director1.safenet editcluster.safenet vhost1.safenet

192.168.1.3             processor1.safenet
192.168.1.4             processor2.safenet

ipvsadm rules for a setup which listens on the director 8080, and load balances the realservers on port 80.

-A -t 192.168.1.1:8080 -s rr
-a -t 192.168.1.1:8080 -r 192.168.1.3:80 -m -w 1
-a -t 192.168.1.1:8080 -r 192.168.1.4:80 -m -w 1

apache: note that I have many virtual hosts, and then one domain for the SSL content.

for reverse proxy cache

<IfModule mod_proxy.c>
        CacheRoot                                               "/tmp/proxy"
        CacheSize                                               1000000 
</IfModule>

for SSL content

<VirtualHost 192.168.0.50:443>
        ServerName editcluster.localnet
        SSLEngine                                               On
        ProxyPass / http://editcluster.safenet:8080/
        ProxyPassReverse / http://editcluster.safenet:8080/
</VirtualHost>

one of these for each virtual host

<VirtualHost 192.168.0.50:80>
        ServerName vhost1.localnet
        ProxyPass / http://vhost1.safenet:8080/
        ProxyPassReverse / http://vhost1.safenet:8080/
</VirtualHost>

Then you need a properly configured apache on your realservers that is set up with virtual hosts for vhost1.safenet and editcluster.safenet, all on port 80.

William Francis 29 Jul 2003 14:46:08

Is it possible to use LVS-DR with https without persistence?

James Bourne james (at) sublime (dot) com (dot) au 30 Jul 2003

It is possible. I made sure that the SSL certificate was available to each realserver/virtual host via an NFS mount. I use a single centralised httpd.conf file across all realservers. For example:

<VirtualHost <VIP>:443>
        SSLEngine               On
        ServerName              servername:443

        DocumentRoot            "/net/content/httpd/vhostname"
        ServerAdmin             [email protected]
        ErrorLog                /net/logs/httpd/vhostname/ssl_error_log
        TransferLog             /net/logs/httpd/vhostname/ssl_access_log
        CustomLog               /net/logs/httpd/vhostname/ssl_request_log "%t
%h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
        SSLCertificateFile      /net/conf/httpd/certs/vhostname.crt
        SSLCertificateKeyFile   /net/conf/httpd/certs/vhostname.key
        SSLCipherSuite
ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

	<Directory />
                Options         None
                AllowOverride   None
                Order           Allow,Deny
                Allow from      a.b.c.d/255.255.255.0 a.b.c.d/255.255.255.0
	</Directory>
</VirtualHost>

/net/logs, /net/conf and /net/content are all NFS mount points.

The downside is that unless you have real signed certificates from Thawte etc. your browser may want to confirm the legitimacy of the certificate presented each time it hits a new realserver. This depends on the load balancing method used.

Hence why the use of persistence is good with https.

Horms

The other reason that persistance is a good idea relates to session resumption. This allows subsequent connections to be set up much faster if an end-user connects to the same realserver. Some Layer 4 Switching implementations allow persistance bassed on session Id for this reason. LVS doesn't do this. And it is a bit hard to put into the current code (when I say a bit, I mean more or less impossible).

For those who are interested, this is how Session IDs are used.

An basic SSL/TLS connection has two main phases, the handshake phase and the data transfer phase. Typically the handshake occurs at the begining of the connection and once it has finished data transfer takes place. The handshake uses asymetric (public key) cryptography, typically RSA, while the data transfer uses symetric cryptography, typially something like DES3. When the sesssion begins the public keys are generated. They are then used to securly transfer the keys that are generated for use with the symetric cryptography that is used for the data transfer.

In a Nutshell the idea is to use slow asymetric cryptography to share the keys required for fast symetric cryptography which is used to transfer the data. Unfortunately the handshake itself is quite slow. Especially for many short connections - as the handshake usually only occurs once its persentage of time for a connection diminishes the longer the connection lasts.

To avoid this problem SessionIDs may be used. This alows an end-user and real-server to identify each other using the SessionID that was issued by the real-server in a previous session. When this occurs an abreviated handhake is used which avoids the more expensive parts of the handshake. Thus making things faster.

Note that using different real-servers will not cause connections that try to use Session IDs to fail. They will just use the slower version of the handshake.

Nicolas Niclausse Jul 30, 2003

Indeed, it will be MUCH slower. I've made a few benchmarks, and https with renegotiation is ~20 times slower.

There is an alternative to persistance: you can share the session IDs on the realservers side with distcache http://distcache.sourceforge.net/ ^[6] .

Christian Wicke Jul 31, 2003

Is load balancing based on the session id extracted from the request possible?

Horms

LVS works at layer 4 so fundamentally it doesn't have the capability to handle session ids.

Cheong Tek Mun

Is it possible to have one domain name for https with two VIPs. For example, the DNS for domain name http:/test.com is 166.166.166.100. I have an LVS with these two VIPs: 166.166.166.100 and 166.166.166.101. Can I have https service on both VIPs?

Horms 21 Dec 2004

Yes.

Joe - I assume test.com is listed in DNS as having two IPs

I've been working with SSL accelerators and have been thinking about using them with LVS. After having Foundry, F5 and Cisco in here for a technical review of their products, I find that LVS has the same basic load balancing functionality. F5 uses BSD Unix, Cisco runs on Linux with ASIC chips, and they have fancy GUI and various additional functionality, but LVS has all the same basic stuff.

	Note
	Note: Red Hat is a Cisco customer, and uses Cisco load balancing and ssl solution in-house (I assume), rather than using their own Red Hat Advanced Server + Piranha + SSL solution of their own which would have helped all us RH Linux Piranha customers.

An SSL accelerator is a piece of software running either on a separate box which is inserted into your data stream, decrypting on the way to the server and then re-encrypting on the way back to the client, or running on a card with its own processor, that's inserted into the server to do the same thing.

In simpler times we put SSL Certificates (Verisign, Thawte, etc.) on the realservers, and let the load balancer route the traffic to the realservers where the SSL decryption is done. https, e-mail protocols smtps, imaps, and pops are all SSL Encryption oriented. Additionally, many new applications today have joined the SSL bandwagon. Now SSL decryption is CPU-intensive and adds actual load on your realservers. Solution? Add more real servers behind your load balancer, right? Yes and no.

Adding additional realservers behind your load balancer to offset SSL decryption load is the intuitive solution, since that is what the purpose of a load balancer is, to allow many realservers, and lower the load to each. However some folks today add what is gernerally called "SSL Accelerator" solution to the mix, and remove as many SSL Certificates and SSL decryption from the realservers and pre-process the SSL Encrypted data stream before it hits the real servers. In short SSL Accelerators decrypt the data, then pass the decrypted data via clear-text (standard) protocols (like http, smtp, imap, pop) to the real servers.

e.g. for https, in a standard server, the decrypted https is sent as clear text to the httpd on port 80. Same for smtps -> smtp, imaps ->imap, pops-> pop, etc.

What happens to ssh? I think because ssh is using its own RSA keys (not SSL) on the server the information is encrypted all the way.

The point of the SSL accelerator is to reduce the load on your server (which is now busy crypting, rather than serving pages), so that it can deliver more pages of data to clients without changing your server setup. However you could do the same thing by beefing up you server (or putting an array of realservers behind an LVS) without any change in software on your servers.

The problem then is one of cost. The Ingrian and Sonicwall SSL boxes are in the multiple 10K$ range. The cards cost less, and other stand-alone units that support fewer protocols (like http/https only) cost a few thousand. Suits feel better spending money when it is available (ie. an SSL accelerator array will insure we don't need to spend more money on 1 or 2 or 3 more realservers in the future). So an SSL accelerator is aimed at non-technical, but financially knowlegable managers, rather than techically competent but financially naive computer people. (When faced with the choice of going to an L7 load-balancer or rewriting the application to be L4 friendly, the suits will go for the slow and expensive L7 load-balancer, while the programmers will re-write the application - the choice depends on what you have at hand that you understand).

You can have an SSL Accelerator without a load balancer, but in order to have an array of them, you want a load balancer. (Another choice is DNS round robin, which several people, including Horms, have found, is not a good way to go).

There are two main ways this passing the baton is done. First, your load balancer can load balance all data steams to a few of these SSL Accelerator units, which decrypt any SSL encrypted data, and themselves load balance all protocols to your realservers. Second, your load balancer can load balance just the SSL encrypted data streams to several SSL Accelerator units, which after they decrypt the data, pass it back to your load balancer as clear-text to be routed by your load balancer as non-encrupted data over standard protocols to your realservers.

SSL Accelerators are available as stand alone units (you would normally buy several and make an array of them) or as SSL Accelerator CARDS which plug into your realservers to speed them up with regard to SSL decryption (which is yet another solution).

In principle you could separate out the SSL handingly from a linux server and run it on a separate box to make a linux only SSL accelerator box. The natural people to do this would be the mod_ssl people, but they are supporting linux compatible SSL hardware (e.g. cards that use the Broadcom chipset) via the "OpenSSL engine" feature. (see the mod_ssl mailing list archives )

This will allow you to build a SSL Accelerator Linux box or to beef up your realserver with a Broadcom card in your Linux LVS load balancer. Since the extra processing is now on a separate processor, in principle this should not add a lot of extra load to your realserver.

Ingrian and Sonicwall are a couple fairly expensive SSL Accelorator solutions which support all the protocols you need. Broadcom makes the card you can add to your real servers. There are other brands which make less expensive SSL Accelerators that support only https (web) protocol.

The information from most vendors is vendor-speak, however Ingrian has some white papers.

I peronally called Ingrian and found them to be extremely TECHNICALLY helpful (more than I could understand myself). They seemed willing to help me even though I told them I use Piranha/LVS (and not Foundry their partner). I did not find any level of open-source style detail on SSL Acceleration/decruption. Got all my info from the load balancer companies, Ingrian, and white papers.

I don't know how an SSL accelerator box/card works. Presumably several levels are involved.

The end result is that the box/card takes the SSL encrypted form of the data, decrypts it to clear text, and shoots it out like it was never encrypted.

Some SSL accelerators have load balancing built-in (eg Ingrian). But not all. The standard "one arm config" does NOT require it. You use your own load balancer, and send SSL encrypted data to your SSL Accelerators, then they output decrypted clear text data sending it back to the load balancer for routing to the realservers. Only Ingrian mentioned/recommended that they can also do load balancing.

I don't know how you would use an SSL accelerator with LVS-DR. The load balancer companies talk about the decrypted clear text going to the realservers, but do not recall a dicussion about going back out again.

Matthias Krauss MKrauss (at) hitchhiker (dot) com 10 Mar 2003 (severly editorialised by Joe)

I had LVS-DR forwarding http (but not https). I hoped to decrypt the https packets on the director with the SSL accelerator card and then pass the decrypted packets to the realserver via the LVS, to save cpu time on realservers. I simulated 10 concurrent requests and downloads of about 3 GB via the ssl acclerator, apache's cpu time went up to 30% on a 1Ghz/512MB host.

when you had the accelerator card in front of the director, what does it do with traffic to other ports eg port 80? Does it just pass them through? Does it look like a router/bridge except for port 443?

For me it looked like some kind of proxy, dealing with the incomming ssl/443/encrypted traffic, decrypts it and passed http/80/decrypted traffic to the LVS . With tcpdump I saw that between the ssl rewrite engine and the VIP was only regular http traffic. By the way, I didnt have an acclerator card for the apache box, I just used apache rewrite and proxy pass mod for the decrytion job.

Julian 11 Mar 2003

The SSL Accelerator cards I know allow user space processes (usually many threads) to accelerate the handling of private keys. What I know is that the normal traffic is still encrypted and decrypted from the CPU(s).

OTOH, decryptying the SSL data in directors is used mostly to modify the HTTP headers for cookie and scheduling purposes. For other applications it can be for another reason. Even the HTTP stream from the realservers is modified. So, I don't think LVS can be used here. Of course, it is possible to implement everything in such way so the user space handling is avoided and all processing is moved in kernel space: queues for SSL async processing, HTTP protocol handling, cookies, just like ktcpvs works in kernel space to avoid memory copy.

I don't follow the SSL forums, so I don't know at what stage is the kernel-level SSL acceleration. My experience shows that the user-space model needs many threads just for private keys to keep the accelerator busy and the rest is spent for CPU encryption and decryption of the data.

Joe: private keys are kept in threads rather than in a table?

Julian: If you have to use the following sequence for an incoming SSL connection (user space):

SSL_accept loop: SSL_read, SSL_write to client read, write to realserver

The SSL_accept operation is the bottleneck for non-hardware SSL processing, SSL_accept handles the private key which costs very much. The hw accel cards offload atleast this processing (the engine is used internally from SSL_accept) but we continue to call SSL_read and SSL_write without using the hw engine.

Considering the above sequence we have two phases which repeat for every incoming connection:

• wait the card drivers to finish the private key operations. That means one thread waiting in blocked state for SSL_accept to finish.
• do I/O for the connection (this includes data encryption and decryption and everything else)

What we want is while the card is busy with processing (it does not have any PCI I/O during this processing) to use the CPU not for the idle kernel process but for encryption and decryption of other connections that are not waiting the engine. So, the goal is to keep the queue of the hwaccel busy with requests and to use the CPU at the same time for other processing. As result, the accel reaches its designed limit of RSA keys/sec and we don't waste CPU in waiting only one SSL_accept for results.

Joe: if there was only one thread, the accelerator would be a bottle neck or it wouldn't work at all?

The CPU is idle waiting SSL_accept (the card) then the card is idle waiting the CPU to encrypt/decrypt data. The result could be 20% usage of the card and (30% usage of the CPU) and the idle process is happy: 70% CPU.

I don't know if this is true for all cards. But even with an accelerator, the using of SSL costs 3-4 times more than just the plain HTTP. My opinion is that this game is useful only for cookie persistence. For other cases LVS can be used in directors and the accelerators on the realservers - LVS forwards TCP(:SSL:HTTP) at L4, the accel is used from user space as usually.

Joe: is this 3-4 times the number of CPU cycles?

Yes, handling of SSL encrypted HTTP traffic with hwaccel is 3-4 times slower than handling the same HTTP traffic without using SSL. Of course, without hwaccel this difference could be 20 and that depends on the used CPU model. What I want to say is that it is better to delay this processing at the place where it is needed: if the SSL traffic needs to be decrypted for scheduling and persistence reasons than it should be done in director but if SSL is used only as secure transport and not for the above reasons than it is better to buy one or more hwaccel cards for the realservers. Loading the director should be avoided if possible.

As for any tricks to include LVS in the SSL processing I don't know how that can be done without using kernel-space SSL hwaccel support. And even then, LVS can not be used, may be ktcpvs can perform URL switching and cookie management.

pb Mar 11, 2003

I wonder if a software engine could be written to accept data from any SSL service (https/smtps/imaps/pops) and let apache rewrite + proxy pass mod decrypt it, then get it sent back out the correct clear text port (http/smtp/imap/pop). Its all SSL encrypted the same way, so once decrypted just pass it to the right protocol. No?

Horms

Yes this would be possible. But I fail to see why it would be desirable unless you are running a daemon that can't do SSL itself (not all demons are SSL enabled).

Joe: see SSL'ifying demons.

unknown: Would an SMP system make a difference such that the OS and general I/O is not bogged down?

Horms

Surely the problem is CPU and not I/O so to that end one would expect that an SMP machine would help.

Ratz

the simple solution that works in almost all cases is apache + mod_reverse_proxy. You can build highly scalable and secure transaction servers. No need for SSL accelerators, since after all, there are only 2 known products out there that work with linux. Also you can double-balance the requests with the same load balancer before the reverse proxy and after the proxy with LVS-DR and two NICs which leads to a pretty nice HA setup. (The healthchecking scripts however are a bit complex though.)

Horms

Personally I agree that SSL accelerators are pointless. I don't really see that you can do much that can't be done with a reverse proxy or better still, just handling the SSL on the realservers. When you think about it, SSL is probably at least as CPU intensive as whatever else the Real Servers are doing, so it makes sense to me that load should be spread out. I can see some arguments relating to SSL session IDs and the like. But the only real way forward here is to move stuff into the kernel. I haven't given this much thought, but what Julian had to say on the matter makes a lot of sense.

(from a while back)

Jeremy Johnson jjohnson (at) real (dot) com 12 Oct 2000

I have been evaluating the Intel 7110 and the 7180 SSL E-commerce accelerators recently. I would prefer to use the 7110's (Straight SSL Accelerators) coupled with LVS instead of throwing out LVS and using the 7180 as the cluster director.

Now, before anyone says "You can do that with LVS-Tun", I am aware that I can fix this problem with LVS-Tun but I would like to see if there is a fix for LVS-DR that would let this work as I prefer the performance of LVS-DR over LVS-Tun. Here is how I have the network setup

________ | | | client | |________| | (router) | __________ | | | | | director |---| |__________| | | | ----------------------------------- | | | | | | ____________ ____________ ____________ | | | | | | | 7110 SSL | | 7110 SSL | | 7110 SSL | |____________| |____________| |____________| | | | | | | ____________ ____________ ____________ | | | | | | | realserver | | realserver | | realserver | |____________| |____________| |____________|

These 7110 SSL Accelerators have 2 Ports, IN one and OUT one. The problem is that when the request comes into the Director and the director looks up the MAC address of the RealServer, it is getting the MAC address of the First NIC in the 7110 Director, so when the packet headers are rewritten with the Destination MAC Address, the MAC address of a NICK in the 7110 is written instead of the MAC of the RealServer. The effect is a black hole.The cluster works fine without the 7110 in between LVS and the RealServer, as soon as I swap in the 7110, all traffic bound for the RealServer with the 7110 in front of it is blackholed.

Any Ideas? I could be wrong about what exactly is happening but basically as soon as I swap in the 7110 LVS for the RealServer Dies, same thing when I set the box in fail-through mode, the box is acting as a straight piece of wire then and I am having the same problem so I believe it has something to do with the wrong MAC address, any ideas?

I am really hoping that someone has encountered something like this and has a fix. The obvious fix would be to be able to on the director force the MAC addresses of the RealServer instead of looking them up.

Terje Eggestad terje (dot) eggestad (at) scali (dot) no 13 Oct 2000

Since you're in an evaluation process, I offer another solution to the accelerator problem, that will give you a cleaner lvs setup. I've tried a PCI accelerator card from RainBow, RainBow. My experience was with Netscape Enterprise Server on HP, but RedHat flags that their version of Apache, Stronghold, has drivers for this card. The effect was nothing short of amazing. Before adding this card the CPU usage ration between the web server and the CGI programms was 75:25. After adding this card it was in the neighbourhood of 10:90.

Ryan Leathers ryan (dot) leathers (at) globalknowledge (dot) com 17 Aug 2005

I have been using LVS for a small farm for a couple of years now. I am interested in adding SSL hardware acceleration to my two LVS servers. It is my goal to maintain performance by offloading the SSL chores, and reduce the cost of certificate renewal by not applying certificates to my web servers.

Can anyone offer advice from experience doing the same? I am using an LVS-NAT configuration currently and am happy with it. It has been suggested that I get a commercial product to do this (Big-IP from F5) which I am not absolutely opposed to, but if there is a good track record with adding SSL hardware acceleration to LVS then I will be happy to stick with what I've been using.

Peter Mueller pmueller (at) sidestep (dot) com

Intel used to make a daisy-chain network device that would do this. A lot of companies still add an SSL card to a few servers, e.g. http://h18004.www1.hp.com/products/servers/security/axl600l/ or http://www.chipsign.com/modex_7000.htm. And then there are the accelerators on F5s and their like. I think the least disruptive way will be the add-on-card to two servers and a :443 vip containing only them.

kwijibo (at) zianet (dot) com

Why are you worrying about offloading it? I would just buy some boxes with faster CPU's if speed is a concern. The time it takes to ship the data back and forth over the bus to the SSL accelerator your processor probably could have taken care of it. Especially if the algorithm uses all the bell and whistle features todays CPUs have.

	Note
	Joe: as Horms says, it's all about the number of certificates. This is not a problem that has a technical solution.

Richard Lyons posting to the qmail mailing list, 05 Feb 2003

To configure Qmail to use "pop3s" and "smtps", there are plenty of examples in the archives at http://www-archive.ornl.gov:8000/, but let me hit the high points.

There are two ways to offer secure versions of the SMTP and POP services. In the first, the existing services on port 25 and 110 can be enhanced with the STARTTLS and STLS extensions (RFCs 2487 and 2595), allowing clients to negotiate a secure connection. The other method is to provide services on different ports and wrap or forward connections on the new ports to the existing services.

STARTTLS for qmail-smtpd is done either with the starttls patch (look for starttls on http://www.qmail.org) or with Scott Gifford's TLS proxy, see http://www.suspectclass.com/~sgifford/stunnel-tlsproxy/stunnel-tlsproxy.html

As far as I know, the only implementation of STLS for qmail-pop3d is Scott's TLS proxy, see the above link.

The starttls patch requires patching and installing a modified qmail-smtpd/qmail-remote but no other changes (apart from configuring certs, etc). The starttls patch also allows secure connections from your mailserver to others supporting RFC2487.

The TLS proxy requires patching and installing a modified stunnel and changing your run scripts, but doesn't modify the current qmail install.

Creating new services can be done with stunnel (http://www.stunnel.org) or sslwrap (http://www.quiltaholic.com/rickk/sslwrap/). You can either configure a daemon to listen on the secure ports (465 and 995 for SSMTP and SPOP3) and forward the traffic to the normal services, or run a service on the ports that invokes the secure wrapper inline. A drawback of the first approach is that connections appear to be from 127.0.0.1, reducing the usefullness of tcpserver on the unencrytped port (I'm told there's a work around for this on Linux machines).

New services requires configuring stunnel and new run scripts but no changes to the existing installation.

An example of stunnel-3 wrapped services can be found in http://www.ornl.gov/its/archives/mailing-lists/qmail/2003/01/msg01105.html

Jesse reports success with stunnel-4 wrapped services in http://www.ornl.gov/its/archives/mailing-lists/qmail/2002/09/msg00238.html

Finally, let me note that if your users want secure services, they should using something like PGP/GPG and APOP.

Brad Taylor btaylor (at) Autotask (dot) com 8 Aug 2005

I'm running a web server and Squid in reverse proxy mode and terminating SSL on the Squid box allowing HTTP to the realserver. I want to add two more real servers for a total of 3. Could I put and LVS into this mix in front of the Squid server and add IP's to Squid and load balancing the Squid IP's allowing Squid to continue to terminate SSL? Traffic would move like this: HTTPS -> (LVS) -> HTTPS (1 of 3 Squid IP's) -> HTTP (1 of 3 real servers). Or would the SSL traffic at the LVS need to decrypted 1st?

Joe 2005-08-08

Whether it's better to have the SSL decryption on the realserver or in a separate SSL accelarator isn't clear AFAIK. It's not like it comes up a lot and we've figured out what to do. Horms probably has the clearest point on the matter which is not to have a separate SSL engine but to have each realserver do its own decrypting/encrypting.

Graeme Fowler graeme (at) graemef (dot) net 08 Aug 2005

I've built a similar system at work (can't go into too much detail about certain parts of it, sadly) but the essence is as follows:

           director + failover director (keepalived/LVS)
                    |
    _________________________________
   |                |                |
squid 1      ... squid 2      ... squid N
   |________________|________________|
   |                |                | 
realserver 1 ... realserver 2 ... realserver N

The Squids are acting in reverse proxy mode and I terminate the SSL connections on them via the frontend LVS, so they're all load balanced. Behind the scenes is a bit more complicated as certain vhosts are only present on certain groups of servers within the "cluster" and the Squids aren't necessarily aware where they might be, so the Squids use a custom-written redirector to do lookups against an appropriate directory and redirect their requests accordingly.

In a position where you have a 1:1 map of squids/realservers you could in theory park a single server behind a single squid, but that doesn't give you much scaleability. It does however mean if a webserver fails then that failure gets cascaded up into the cluster more easily. Then again, if you're shrewd with your healthchecks you can combine tests for your SSL IP addresses and take them down if all the webservers fail.

Also, don't overallocate IP addresses on the Squids. If I say "think TCP ports", you only need a single IP on your frontend NIC... but I'll leave you to work that one out!

Remember that the LVS is effectively just a clever router; it isn't application-aware at all. That's what L7 kit is for :)

Horms

Here is a description I wrote a while ago about SSL/LVS In a nutshell, you probably want to use persistance and have the real-servers handle the SSL decryption. http://archive.linuxvirtualserver.org/html/lvs-users/2003-07/msg00184.html Actually, using the lblc scheduler, or something similar, might be another good solution to this problem.

Graeme

...this is OK if you have a set of servers onto which you can install multiple SSL Certs and undergo the pain of potentially having an IP management nightmare:

1 realserver, 1 site -> 1 "cluster" IP
2 realservers, 1 site -> 2 "cluster" IPs
...
10 realservers, 10 sites -> 100 "cluster" IPs

I think you can see where this is going! Very rapidly the IP address management becomes unwieldy.

You can of course get around this by assigning different *ports* to each site on each server, but if you have the cert installed on all servers in the cluster this soon becomes difficult to manage too.

Offloading onto some sort of SSL "proxy", accelerator, engine (call it what you will) means that you can then simply utilise the processing power of that (those) system(s) to do the SSL overhead and keep your webservers doing just that, serving pages. If each VIP:443 points to a different port on the "engine" you greatly simplify your address management too.

Horms

I do not follow how having traffic that arrives on the real-servers in plain-text or SSL changes the IP address situation you describe above.

Graeme

Also, most commercial SSL certification authorities will charge you an additional fee to deploy a cert on more than one machine, so if you can reduce the number of "engine" servers you reduce your costs by quite some margin.

Horms

This is the sole reason to use an SSL accelerator in my opinion. The fact is that SSL is likely to be the most expensive ($) operation your cluster is doing. And that in many cases it is cheaper to buy some extra servers, and offload this processing to an LVS cluster, than to by an SSL accelerator card.

Graeme

You do however still need to use persistence, and potentially deploy some sort of "pseudo-persistence" in your engines too to ensure that they are utilising the same backend server. If you don't, you'll get all sorts of application-based session oddness occurring (unless you can share session states across the cluster).

Horms

If you have applications that need persistence, then of course you need persistence regardless of SSL or not. But if you are delivering SSL to the real servers then you probably want persistence, regardless of your applications, to allow SessionID to work, and thus lower the SSL processing cost.

	Note
	summary: this doesn't work

An alternative to propagating updates to databases is to use a distibuted filesystem, which then becomes responsible for the updates appearing on other machines. (see Filesystems for realserver content and Synchronising realserver content). While this may work for a webserver, a databased is not happy about having agents outside its control writing to its data filesystem.

A similar early naive approach approach of having databaseds on each realserver accessing a common filesystem on a back-end server, fails. Tests with mysqld running on each of two realservers working off the same database files mounted from a backend machine, showed that reads were OK, but writes from any realserver either weren't seen by the other mysqld or corrupted the database files. Presumably each mysqld thinks it owns the database files and keeps copies of locks and pointers. If another mysqld is updating the filesystem at the same time then these first set of locks and pointers are invalid. Presumably any setup in which multiple (unlocked) databaseds were writing to one file system (whether NFS'ed, GFS'ed, coda, intermezzo...) would fail for the same reason.

In an early attempt to setup this sort of LVS jake buchholz jake (at) execpc (dot) com setup an LVS'ed mysql database with a webinterface. LVS was to serve http and each realserver to connect to the mysqld running on itself. Jake wanted the mysql service to be lvs'ed as well and for each realserver to be a mysql client. The solution was to have 2 VIPs on the director, one for http and the other for mysqld. Each http realserver makes a mysql request to the myqslVIP. In this case no realserver is allowed to have both a mysqld and an httpd. A single copy of the database is nfs'ed from a fileserver. This works for reads.

Malcolm Cowe

What if the realservers were also configured to be part of an HA cluster like SGI Failsafe or MC/ServiceGuard (from HP)? Put the real servers into a highly available configuration (probably with shared storage), then use the director to load balance connections to the virtual IP addresses of the HA cluster. Then you have a system that parallelises the database and load balances the connections. This might require ORACLE Open Parallel Server (OPS) edition of the database, you'd have to check.

"John P. Looney" john (at) antefacto (dot) com 12 Apr 2002

My previous employer did try such a system, and had very little luck with it. In general, you are much better off having one writer, and multiple readers. Then, update the "read-only" databases from the write-database. We were using Oracle Parallel Server on HP-UX, and it crashed about once every two weeks, not always just under heavy load. Databases are by design single entities. Trying to find a technical solution to such a mathematical problem is asking for pain and suffering, over a prolonged period of time.

	Note
	May 2001: The LSD project does not seem to be active anymore. The replication feature of mysql is functionally equivelant.

The Linux Scalable Database (LSD) project http://lsdproject.sourceforge.net/ is working on code to serialise client writes so that they can be written to all realservers by an intermediate agent. Their code is experimental at the moment, but is a good prospect in the long term for setting up multiple databased and file systems on separate realservers.

Gaston Gorosterrazgoro (at) hostarg (dot) com (dot) ar Jun 12 2003

I solved my problems quite different than seen in the HOW-TO. So, someone may need this email in the near future:

-------------------------------------------------------------------------
    Director
        192.168.0.23 (eth1)
        192.168.0.24 (eth0:100)
        10.0.0.1 (eth0)
        Just configured ip_vs to load balance Apache between Server3 and
Server1 (in this order because for me, Server3 is the primary apache server)
-------------------------------------------------------------------------
    Server1 (primary MySQL, secondary Apache)
        10.0.0.3 (eth0)
        10.0.0.10 (eth0:100)
        MySQL is Master/slave for Server3

-------------------------------------------------------------------------
    Server2 (secordary MySQL)
        10.0.0.4 (eth0)
        Mon running, checking every 10 secs. MySQL in 10.0.0.3. If it fails,
then this machine configures eth0:100 as 10.0.0.10.
        MySQL is Master/slave for Server1
-------------------------------------------------------------------------
    Server3 (primary Apache)
        10.0.0.5
        Apache (in my case php), connects always to 10.0.0.10.
-------------------------------------------------------------------------

End of story. I have better security in the MySQL daemons (not accesible from clients), less charge in the director machine (don't have to worry about MySQL, neither run MON yet), and I'm happy. :) Now is the turn for Cyrus in Server2 and Server3.

There is a writeup (Mar 2006) at http://www.howtoforge.com/loadbalanced_mysql_cluster_debian by Falko Timme based on UltraMonkey. (we don't know this guy and he's never posted to the LVS mailing list - he seems to know what he's doing though.)

MySQL (and most other databases) supports replication of databases.

Ted Pavlic tpavlic (at) netwalk (dot) com 23 Mar 2001

When used with LVS, a replicated database is still a single database. The MySQL service is not load balanced. HOWEVER, it is possible to put some of your databases on one server and others on another. Replicate each SET of databases to the OTHER server and only access them from the other server when needed (at an application or at some fail-over level).

Doug Sisk sisk (at) coolpagehosting (dot) com 9 May 2001

An article on mysql's built in replication facility

Michael McConnell michaelm (at) eyeball (dot) com> 13 Sep 2001

Can anyone see a down side or a reason why one could not have two Systems in a Failover Relationship running MySQL. The Database file would be synchronized between the two systems via a crontab and rsync. Can anyone see a reason why rsync would not work? I've on many occasions copied the entire mysql data directory to another system and start it up without a problem. I recognize that there are potential problems that the rsync might take place while the master is writing and the sync will only have part of a table, but mysql's new table structure is supposed to account for this. If anything, a quick myismfix should resolve these problems.

Michael McConnel michaelm (at) eyeball (dot) com 2001-09-14

There are many fundamental problems with MySQL Replication. MySQL's Replication requires that two systems be setup with identical data sources, activate in a master/slave relationship. If the Master fails all requests can be directed to the Slave. Unfortunately this Slave does not have a Slave, and the only way to give it a slave, is to turn it off, synchronize it's data with another system and then Activate them in a Master/Slave relationship, resulting in serious downtime when databases are in excess of 6 gigs (-:

This is the most important problem, but there are many many more, and I suggest people take a serious look at other options. Currently I use a method of syncing 3 systems using BinLog's.

Paul Baker pbaker (at) where2getit (dot) com>

What is the downtime when you have to run myisamchk against the 6 gig database because rsync ran at exactly the same time as mysql was writting to the database files and now your sync'd image is corrupted?

There is no reason you can not set up the slave as a master in advance from the beginning. You just use the same database image as from the original master.

When the master master goes down, set up a new slave by simple copying the original master image over to the new slave, then point it to the old slave that was already setup to be a master. You wouldn't need to take the original slave down at all to bring up a new one. You would essentially be setting up a replication chain but only with the first 2 links active.

Michael McConnell michaelm (at) eyeball (dot) com>

In the configuration I described using Rsync, the MyISMchk would take place on the slave system, I recognize the time involved would be very large, but this is only the slave. This configuration would be setup so an Rsync between the master and slave takes place every 2 hours, and then the Slave would execute a MyISMchk to ensure the data is ready for action.

I recognize that approximately 2 hours worth of data could be lost, but I would probably us the MySQL BinLogs rotated at 15 minutes interval and stored on the slave to allow this to be manually merged in, and keep the data loss time down to only 15 minutes.

Paul, you said that I could simply copy the Data from the Slave to a new Slave, but you must keep in mind, in order to do this MySQL requires that the Master and Slave data files be IDENTICAL, that means the Master must be turned off, the data copied to the slave, and then both systems activated. Resulting in serious downtime.

Paul

You only have to make a copy of the data one time when you initial set up your Master the first time. As long as it takes to do this is your downtime:

kill mysqlpid tar cvf mysql-snapshot.tar /path/to/mysql/datadir /path/to/mysql/bin/safe_mysqld

Your down time is essentially only how long it takes to copy your 6 gigs of data NOT across a network, but just on the same machine. (which is far less than a myisamchk on the same data) Once that is done, at your leisure you can copy the 6 gigs to the slave while the master is still up and serving requests.

You can then continue to make slave after slave after slave just by copying the original snap shot to each one. The master never has to be taking offline again.

Michael McConnell

You explained that I can kill the MySQL on the Master, tar it up, copy the data to the Slave and activate it as the Slave. Unfortunately this is not how MySQL works. MySQL requires that the Master and Slave be identical data files, *IDENTICAL* that means the Master (tar file) cannot change before the Slave comes online.

Paul

Well I suppose there was an extra step that I left out (because it doesn't affect the amount of downtime). The complete migration steps would be:

1. Modify my.cnf file to turn on replication of the master server. This is done while the master mysql daemon is still running with the previous config in memory.
2. shutdown the mysql daemon via kill.
3. tar up the data.
4. start up the mysql daemon. This will activate replication on the master and cause it to start logging all changes for replication since the time of the snapshot in step 3. At this point downtime is only as long as it takes you to do steps 2, 3, and 4.
5. copy the snapshot to a slave server and active replication in the my.cnf on the slave server as both a master and a slave.
6. start up the slave daemon. at this time the slave will connect to the master and catch up to any of the changes that took place since the snapshot.

So as you see, the data can change on the master before the slave comes online. The data just can't change between when you make the snapshot and when the master daemon comes up configured for replication as a master.

Michael McConnell

Paul you are correct. I've just done this experiment.

A(master) -> B(slave)
B(master -> C(slave)

A Died. Turn off C's Database, tar it up, replicated the Data to A, Activate A as Slave to C. No data loss, and 0 downtime.

(there appears to have been an offline exchange in here.)

Michael McConnell michaelm (at) eyeball (dot) com>

I've just completed the Rsync deployment method, this works very well. I find this method vastly superiors to both other methods we discussed. Using Rsync allows me to only use 2 HOSTS and I can still provide 100% uptime. In the other method I need 3 systems to provide 100% uptime.

In addition the Rsync method is far easier to maintain and setup.

I do recognize this is not *perfect* I run the Rsync every 20 minutes, and then I run myismchk on the slave system immediately afterwards. I run the MyISMChk to only scan Tables that have changed since the last check. Not all my tables change every 20 minutes. I will be timing operations and lowering this Rsync down to approximately 12 minutes. This method works very effectively for managing a 6 Gig Database that is changing approximately 400 megs of data a day.

Keep in mind, there are no *real time* replication methods available for MySQL. Running with MySQL's building Replication commonly results (at least with a 6 gig / 400 megs changing) in as much as 1 hour of data inconsistency. The only way to get true real time is to use a shared storage array.

Paul Baker

MySQL builtin replication is supposed to be "realtime" AFAIK. It should only fall behind when the slave is doing selects against a database that causes changes to wait until the selects are complete. Unless you have a select that is taking an hour, there is no reason it should fall that far behind. Have you discussed your findings with the MySQL developers?

Michael McConnell

I do not see MySQL making any claims to Real-time. There are many situations where a high load will result in systems getting backed up, especially if your Slave system performs other operations.

MySQL's built-in replication functions like so;

Alexander N. Spitzeraspitzer (at) 3plex (dot) com

how often are you running rsync? since this is not realtime replication, you run the risk of losing information if the master dies before the rsync has run (and new data has been added since the last rsync.)

Don Hinshaw dwh (at) openrecording (dot) com>

I have one client where we do this. At this moment ( I just checked) their database is 279 megs. An rsync from one box to another across a local 100mbit connections takes 7-10 seconds if we run it at 15 minute intervals. If we run it at 5 minute intervals it takes <3 seconds. If we run it too often, we get an error of "unexpected EOF in read_timeout".

I don't know what causes that error, and they are very happy with the current situation, so I haven't spent any time to find out why. I assume it has something to do with write caching or filesystem syncing, but that's just a wild guess with nothing to back it up. For all I know, it could be ssh related.

We also do an rsync of their http content and httpd logs, which total approximately 30 gigs. We run this sync hourly, and it takes about 20 minutes.

Benjamin Lee benjaminlee (at) consultant (dot) com>

For what it's worth, I have also been witness to the EOF error. I have also fingered ssh as the culprit.

John Cronin

What kind of CPU load are you seeing? rsync is more CPU intensive than most other replication methods, which is how it gains its bandwidth efficiency. CPU Load? How many files are you syncing up - a whole filesystem, or just a few key files? From you answer, I assume you are not seeing a significant CPU load.

Michael McConnell

I RSync 6 Gigs worth of data, Approximately 50 files (tables). Calculating Checksum's is really a very simple calculation, the cpu used to do this is less than 0% - 1% of a PIII 866. (care of vmstat)

I believe all of these articles you have found are related to RSync Servers that serve in the function one would see a system as a major FTP Server. For example ftp.linuxberg.org or ftp.cdrom.com

Mark

We're using master/slave replication. The LVS balances the reads from the slaves and the master isn't in the load-balancing pool at all. The app knows that writes go to the master and reads go the VIP for the slaves. Aside from replication latency, this works very reliably.

Todd Lyons tlyons (at) ivenue (dot) com 26 Oct 2005

This works for us too. We have an inhouse DBI wrapper that knows to read from slaves and write to the master. Reads are directed to a master for a certain number of seconds until we are reasonably sure that the data has been replicated out.

this is exactly how i was doing it. The problem was that i expected the data on the slaves to be realtime - and the replication moved too slow. not to mention it broke a lot (not sure why) - anytime there was a query that failed it halted replication.

We've not seen this. Our replication has been very stable, and our system is roughly 80% read, 20% write. But we do as few reads as possible on the master, keeping all of the reads on the slaves, doing writes to the master.

in which case, i had to either take the master down, or put it in read-only mode, so i could copy the database over, and re-initialize replication.

Make a tarball of the databases that you replicate and keep that tarball someplace available for all slaves. Record the replication point and filename. Then when you have to restart a slave, blow away the db, extract the tarball, configure the info file to the correct data, then start mysql and it will populate itself from the bin files on the master. For us, we try to do it once per month, but I've seen 100 day bin file processing and it only took about 30 minutes to catch the database up. We have gig networking though, so your results may vary.

We also have a script that connects and checks the replication status for each slave and spits out email warnings if the bin logs don't match. It doesn't check position because it frequently changes mid-script, sometimes more than once. Here's an example:

[todd@tlyons ~]$ checkmysqlslave.sh 
Status Check of MySQL slave replication.
Log file names should match exactly, and
log positions should be relatively close.

Machine      : sql51
Log File     : fs51-bin.395
Log Position : 18123717

Machine      : sql52
Log File     : fs51-bin.395
Log Position : 18123863

Machine      : images3
Log File     : fs51-bin.395
Log Position : 18123863

Machine      : images4
Log File     : fs51-bin.395
Log Position : 18124769

Machine      : images5
Log File     : fs51-bin.395
Log Position : 18125333

Machine      : images6
Log File     : fs51-bin.395
Log Position : 18125333

of course, this was on sub-par equipment, and maybe nowadays it runs better. i'm thinking that the NDBcluster solution might be better since it's designed for this more - replication still (afaik) sucks because when you need to re-sync it requires the server to be down or put in read-only, which is essentially downtime. The main problem I see with NDB is that it requires the entire database (or whatever part each node is reponsible for) to be in memory.

We have had very stable results from mysql stable releases doing replication. We typically use the mysql built rpms instead of the vendor rpms. We had serious stability problems with NDB, but we were also trying a very early version. It is liable to be much more solid now.

not work with files the way normal InnoDB, MyIsam, etc engines work. And I think this is still the case in MySQL 5 (correct me if I'm wrong.) I don't know what happens if your storage node suddenly runs out of Memory.

It won't be until MySQL 5.1 that you'll be able do file based clustering. Then load balancing becomes simple, for both reads and writes.

Matthias Mosimann matthias_mosimann (at) gmx (dot) net 2005/03/01

I also don't have experience with any of these solutions but...

Jan Klopper janklopper (at) gmail (dot) com 2005/03/02

Your best method (if you only do a lot of Select query's) would be to use one mysql master server which gets all the update/delete/insert query's and stores the main database,. Then you could use mysql master-slave setup to replicate this master database to a couple of slave servers on which you could do all of your selects.

You could use Mon/ldirectord to make a LVS cluster out of these select servers just like any other type of servers. I'm not sure what you should do with the persistence though (especially if you use mysql_pconnect from within php/apache).

I have the following setup:

Granted, if your database server dies, you're screwed, but with the current mysql master/slave/replication/cluster technologies you wouldn't be able to provide failover for update/insert/delete query's anyway.

	Note
	making one of the slaves into a master and thus change its database will give you a serious problem when you master comes back up again. You'd have to make the master a slave, stop the current master (breaking the whole point of allways online) and replicate to it, then when it is up2date change its role to be master again.

Jan Klopper

So you can setup a mysql server on each of the nodes of you cluster (allowing all of the nodes to be load balanced mysql servers)

What would be a better sollution (afaik) is to place mysql on all the apache servers as well, since this would would allow them to connect to their own database instead of using a network connect, decreasing the overhead for each query/connect. And if one of the webserver nodes went down, so does its mysql, so thats not a real problem. LVS will just not redirect any request to the machine at all. A not clustered server would then be the nbdd server managing and updating all of the mysql servers on the nodes itself. And thus you could use really inexpensie hardware for the mysql server, as you won't need UBER speed on that machine to handle all requests from all LVS-nodes anymore.

Todd Lyons tlyons (at) ivenue (dot) com

We've always found that separating services out to different machines works better than running multiple services on each machine.

Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) 123multimedia (dot) com (replying to Jan)

Well you still need a very powerfull nbdd server, with high network capabilities, and you should use at least two data nodes.

Matthias Mosimann

Here's a thread in the mysql mailing list about clustering / loadbalancing mysql http://forums.mysql.com/read.php?25,14754,14754#msg-14754

Francois JEANMOUGIN

The idea is to have small MySQL nodes on the realservers and 2 or 3 data nodes (powerfull machines) on the backend. The current state of the artr for MySQL is to use an NBD cluster. You can use LVS to load-balance connections to your MySQL servers. You need a small management server to manage it.

Todd Lyons tlyons (at) ivenue (dot) com 2005/03/02

We worked for a short while with NBD clustering and experienced huge problems. It was probably more an issue with perl-DBD instead of NBD itself, but it was enough that we had to go back to standard replication. Our current setup uses a homegrown DBI wrapper that does all reads from the slave and directs all writes to the master (and for a short time all reads in that session go to that master too). As read load escalates, just add another slave and put it on an LVS VIP.

Francois JEANMOUGIN

To check whether a MySQL server is alive or not, you have two solutions (assuming you choose Keepalived):

Jan Klopper

But as soon as you start doing updates to the database, you will horribly screw up your data, because mysql can't propagate the changes made on slaves back to the master and so on. This will lead to inconsistency's in your database. If you only need to do load (and I mean loads) of select queries with no changes than you could very well use mysql slaves inside a LVS to run these queries.

Francois JEANMOUGIN

Are we talking about the same thing? I'm not talking about master/slave or active-slave method using old MySQL replication process. I'm talking about NBD cluster: http://dev.mysql.com/tech-resources/articles/mysql-cluster-for-two-servers.html, http://dev.mysql.com/doc/mysql/en/mysql-cluster-basics.html I'm currently evaluating this solution, which as passed the "case study" test. I will implement it at the end of this month. I would like to know if anyone here have any (bad or good) experience with such MySQL setup.

Todd Lyons tlyons (at) ivenue (dot) com

We've had experience and it was bad. Things like 'DESCRIBE table' would give erratic results. Required a beta version of perl-DBD. Performance was good though. We had a 2 node cluster with plans to scale upwards. I think once supporting code stabilizes it will be a good solution. I like the way they keep things in memory because RAM is cheap. We are also evaluating the true clustering solution provided by Emic Networks. It is fantastic in my opinion, but it's not cheap. It requires custom configuration of your switches and routers to handle multicast MACs, a virtual gateway, and some VLAN'ing. It works well though and scales well. If your system has more than 50% reads, it will work well for you.

Jan Klopper (replying to Francois)

You are talking about pure failover only right? I was talking about load balancing for mysql which is pretty much undoable. Pure failover might be possible, just make sure the slave becomes the master on failover, and when the old master get back up it starts being a slave, and replicates from the new master.

Francois

No no no. Please read the links I sent, and add this one : http://dev.mysql.com/doc/mysql/en/mysql-cluster-overview.html I'm talking about pure active-active cluster. This setup could be strenghted by LVS and a Keepalived MISC checker. In the figure above, you put LVS between Applications and SQL. About the link above, note that mysqld server and nbdd server can be on the same machine.

Jan Klopper

Even normal master-slave replication still gives loads of problems, so anything cutting edge which might do the trick and allow for the slaves to be full database servers (and thus capable of handling updates/inserts/deletes) should not be trusted in my opinion. Granted, if they get the slaves to also be masters to each other, and they get it to pay nice, then we're all set to have the best and fastest clustering database on the planet (except for oracle?).

Bosco Lau bosco (at) musehub (dot) com

Lets forget about 'multi'-master scenario. It is basically an unsolvable problem for MySQL (at the moment). The only sensible multi-server setup for MySQL is 'single master' - 'mutiple slave', preferbly in 'star shape formation' i.e. each slave connected directly to master. Also don't bother about 'daisy-chaining' the slaves, this creates a single point of failure. With the mutiple slaves/LVS setup , you can just use MYISAM tables in the slaves database whereas you may have innoDB tables in the master. This will make your SELECT performances even better. Since the slaves just replicates the data from the master, there is no need for transactional tables type in the slave database.

Jan Klopper

Multi-master setups are a huge problem in mysql. If you do use slaves for just the queries, use MyIsam tables (preferrable stored completely in memmory) to increase select speeds. I'm using mysql to run huge banner/advertisement systems, and thus i need at least one update/insert per view. or i should store them outside of the db, and thus i can;t use normal slaves. (because i'd still need to connect to the master to make the update.) Star formation is the way to go, unless you have a huge amount of changes on the master which make you need a seccond cascade in there to load balance the updates and release the master from most of the slave replication querys. I think this step could also be done with LVS (having a couple of slave's to retreive the updates from one master, balanced by LVS, ) and then use them as masters for a couple of other normal query slaves.

Matthias Mosimann matthias_mosimann (at) gmx (dot) net 2005/04/18

In a mysql cluster, data is stored in memory. If you store about 100 GB in your databaes you need about 100 GB of Memory (ram + swap). Now it looks that there will be a solution for that. I found a thread in the myslq-cluster mailing list and a link in the manual. Mysql 5.1 will be able to handle disk-based records in the cluster.

Thread:
http://forums.mysql.com/read.php?25,20801,20801#msg-20801

Link:
http://dev.mysql.com/doc/mysql/en/mysql-5-1-cluster-roadmap.html

Nigel Hamilton nigel (at) turbo10 (dot) com 2005/04/18

The RAM requirements of the initial MySQL Cluster design makes it unworkable for me so a hybrid RAM + disk-based system is welcome news. Our writes may equal or exceed reads - I was planning on using "memcached" to help with handing off as many reads as possible to a distributed RAM bucket. But I'm still thinking of ways to distribute writes - ideally at the application level all I need is a database handle that connects to one big high speed amorphous distributed DB. My current short term solution to distributing writes is to use a MySQL heap/ram table on each node which acts as a bucket collecting writes which is periodically emptied to the Master's disk.

Matthias Mosimann matthias_mosimann (at) gmx (dot) net

IIRC we had a discussion (Jan Klopper, Francois Jeanmougin and me) on the 2nd March this year on the same topic allready. Here's the link: http://archive.linuxvirtualserver.org/html/lvs-users/2005-03/mail4.html

Francoise Jeanmougin

We made some other testing on high availabilty/high performance MySQL:

We didn't had time to test Ndb properly on our environment, the solution seems to be good in terms of design, it is memory based, and uses table patitionning (so it'll split the datas on several servers). It has to be improved to be really usable and strong.

Troy Hakala troy (at) recipezaar (dot) com 25 Oct 2005

We're using master/slave replication. The LVS balances the reads from the slaves and the master isn't in the load-balancing pool at all. The app knows that writes go to the master and reads go the VIP for the slaves. Aside from replication latency, this works very reliably.

The problem then is failure of the master. The master is redundant in itself (RAID drives, redundant power) to minimize the risk. But yes, we are very read-intensive and keeping the master out of the read load-balancing further increases the master's lifespan.

We haven't tried master-master replication, but it's pretty easy and quick to turn a slave into a master. We prefer simplicity to complexity. Besides, we're not a bank and no one dies if the master db goes down for a few minutes every 5 years. :-)

FWIW, our outages have been caused by bandwidth outages more than server hardware failure. There's supposed to be redundancy there too, but for some reason or another, the redundancy never kicks in. ;-)

Replication can be restarted easily: slave start after you fix the error, which is usually just a mysqlcheck on the table. You shouldn't have to take the master down at all.

And if you use a replication check in LVS, LVS will take the db server out of rotation if it's not replicating, so it shouldn't even be noticed until you can get around to fixing it. Nagios is also recommended to let you know when a slave is no longer replicating.

The latency is on the order of a couple seconds and it's easy to take care of in the app. In fact, it's only a problem if you cache db results for hours or days (we use memcached). So it's not much of a problem in reality if you account for it.

NDB violates my simplicity+commodity ideals... it's complicated and requires very expensive (lots of RAM) servers. And, I think it *requires* SCSI drives for some reason (I thought it was memory-based)! Doesn't Yahoo use MySQL replication? If it's good enough for Yahoo, it's good enough for most people. :-)

If the master went down, we'd make a slave into a master manually. But it's never happened in real life, just in lab tests. A script could be written to do it, I guess, but it's not that simple.

All the talk about redundancy and high-availability is a bit academic, IMO, unless it really is mission-critical -- and if it was, I'd be using Oracle and not MySQL so I could blame them. ;-)

As I mentioned, none of this matters if bandwidth and electricity go out, which is less reliable in my experience than solid-state computer hardware.

mike mike503 (at) gmail (dot) com 25 Oct 2005

perhaps changing a slave to a master has changed from when I used it - anytime a slave died, it would not start replicating (or you wouldn't want to) unless it caught up to the master since it died. but to grab a snapshot of the master data, you would have to take the master down or flush tables with a read lock until it was done getting the data synced. then you could restart it...

for example, i was using this on a forum - when someone posts a message:

webserver inserts data into master master replicates to slave webserver reads from slave

if that doesn't happen within milliseconds, the user is taken to a page that doesn't have their post show up yet.

Pierre Ancelot pierre (at) bostoncybertech (dot) com 26 Oct 2005

I run NDB and it works pretty fine... Debian GNU/Linux Sarge Mysql 4.1 2 nodes and a management server.

Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) 123multimedia (dot) com 26 Oct 2005

Note that it is unable to handle more than about 1000rq/s. Our MySQL server (standalone MyISAM) goes up to 3000rq/s. InooDB hangs at 1024rq/s.

Matthew Enger menger (at) outblaze (dot) com 1 Sep 2000

I am looking into running two NFS servers, one as a backup for serving lots of small text files and am wondering what would be the best solution for doing this. Does anyone have any recomendations.

Joe

LVS handles NFS just fine (see the performance document on the docs page of the lvs website). You have to handle the problem of writes yourself to keep both NFS servers in sync.

Jeremy Hansen jeremy (at) xxedgexx (dot) com 1 Sep 2000

Can nfs handle picking up a client machine in a failover situation. If my primary nfs server dies, and my secondary takes over the first, can nfs clients handle this? Something tells me nfs wouldn't be very happy in this situation.

Wayne wayne (at) compute-aid (dot) com 01 Sep 2000

It will not work for NFS. NFS is working based on server hand over an opaque packet to the client. Client then will communicate with the server based on that opaque handle. Normal NFS construct that opaque handle involves some file system ID from that particular server, which most likely will be different from one server to the other.

Joe

this is a problem even for UDP NFS? (I know NFS is supposed to be stateless but isn't)

Wayne

If the NFS servers are identical, there is a chance that it may work. However, if the file systems are not identical (from file system ID point of view), it will not work, no matter if it is UDP or not. The stateless is only true to that particular server. BUT, that is the NFSes I had been worked on before based on Sun's original invention, it may not true for others implementations.

Alan Cox alan (at) lxorguk (dot) ukuu (dot) org (dot) uk 1 Sep 2000

IFF your NFS servers are running some kind of duplexing protocol and handling write commits to both disk sets before acking them then the protocl is good enough - for any sane performance you would want NFSv3 The implementations are another matter

John Cronin jsc3 (at) havoc (dot) gtf (dot) org

It works as long as the filesystems are identical. That means either readonly content dd'd to identical disk on failover machine, or dual ported disk storage with two hosts (or more) attached. When the failover happens the backup system then mounts the disks and takes over. You need to be VERY sure that the primary system is NOT up and writing to the disk or you will have to go to backups after the disk gets corrupted (having two systems perform unsynchronized writes to the same filesystem is not a good idea).

This is the way Sun handles it in their HA cluster. They go a step further by using Veritas Volume Manager, and Veritas has to forcibly import the disk group to the backup when a failover is done - the backup also sends a break or poweroff command to the primary via the serial terminal interface to make darn sure the primary is down. That said, I have seen three systems all mounting the same volume during some pathological testing I did on some systems at a Sun HA training course. The storaage used in this situation was Sun D1000/A1000 (dual ported UltraSCSI) and Sun A5000 (Fiberchannel).

Joe

Would the client get a stale file handle on real-server failure? Now that I think about it, I snooped on a tread with similar content recently. It would have been linux-ha or beowulf. I had assumed you could fail out an NFS server, since file servers like the Auspex boxes use NFS (I thought) and can failover. How they work at the low level I don't know.

Joe

in principle this could be handled by generating the filehandle from the path/filename which would be the same on all systems?

Wayne

It could be done like that. But if the hard drive SCSI ID is different that could cause system uses different file system ID assigned to the file system, and end up stalled handle.

Here we're asking one of the authors of the Linux NFS code.

Joseph Mack

One of the problems with running NFS as an LVS'ed service (ie to make an LVS fileserver), that has come up on this mailing list is that a filehandle is generated from disk geometry and file location data. In general then the identical copies of the same file that are on different realservers will have different file handles. When a realserver is failed out (e.g. for maintenance) and the client is swapped over to a new machine (which he is not supposed to be able to detect), he will now have an invalid file handle.

Is our understanding of the matter correct?

Dave Higgen dhiggen (at) valinux (dot) com 14 Nov 2000

In principle. The file handle actually contains a 'dev', indicating the filesystem, the inode number of the file, and a generation number used to avoid confusion if the file is deleted and the inode reused for another file. You could arrange things so that the secondary server has the same FS dev... but there is no guarantee that equivalent files will have the same inode number; (depends on order of file creation etc.) And finally the kicker is that the generation number on any given system will almost certainly be different on equivalent files, since it's created from a random seed.

If so is it possible to generate a filehandle only on the path/name of the file say?

Well, as I explained, the file handle doesn't contain anything explicitly related to the pathname. (File handles aren't big enough for that; only 32 bytes in NFS2, up to 64 in NFS3.)

Trying to change the way file handles are generated would be a MASSIVE redesign project in the NFS code, I'm afraid... In fact, you would really need some kind of "universal invariant file ID" which would have to be supported by the underlying local filesystem, so it would ramify heavily into other parts of the system too...

NFS just doesn't lend itself to replication of 'live' filesystems in this manner. It was never a design consideration when it was being developed (over 15 years ago, now!)

There HAVE been a number of heroic (and doomed!) efforts to do this kind of thing; for example, Auspex had a project called 'serverguard' a few years ago into which they poured millions in resources... and never got it working properly... :-(

Sorry. Not the answer you were hoping for, I guess...

It seems that the code which calculates the filehandle in NFS is so entrenched in NFS, that it can't be rewritten to allow disks with the same content (but not neccessarily the same disk geometry) to act as failovers in NFS. The current way around this problem is for a reliable (eg RAID-5) disk to be on a shared scsi line. This way two machines can access the same disk. If one machine fails, then the other can supply the disk content. If the RAID-5 disk fails, then you're dead.

John Cronin jsc3 (at) havoc (dot) gtf (dot) org 08 Aug 2001

You should be able to do it with shared SCSI, in an active-passive failover configuration (one system at a time active, the second system standing by to takeover if the first one fails). Only by using something like GFS could both be active simultaneously, and I am not familiar enough with GFS to comment on how reliable it is. If you get the devices right, you can have a seamless NFS failover. If you don't, you may have to umount and remount the file systems to get rid of stale file handle problems. For SMB, users will have to reconnect in the event of a server failure, but that is still not bad.

Salvatore J. Guercio Jr sguercio (at) ccr (dot) buffalo (dot) edu 12 Jun 2003

Here is background on my LVS/NFS setup:

I have 4 IBM x330 and an IBM FastT500 SAN, this project I am working is to set up the SAN to server 635GB worth of storage space to the Media Studies department of the University. Each x330 is connected to the SAN with GB fibre channel and 100MB to a Cisco 4006 Switch. The Switch has a GB connection to the media studies department where they have a lab of 4 Macs running MacOS X. They will mount the share and access media data on the share. Each client has a GB connection to the Cisco switch.

The LVS comes in handy to increase the bandwidth of the SAN as well as gives us some redundancy. Since I have a shared SCSI solution, I will not run into the file handle problem and I am hoping that using IBM GPFS filesystem, will help me with the lockd problem.

Did you have to do anything as far as forwarding your portmap or any other ports to the realservers? Right now I am only forwarding udp port 2049.

Joe

Just a sanity check here... Do you understand the problems of exporting NFS via LVS (see the HOWTO)? Are these problems going to affect you (if not, this would be useful for us to know). I assume this setup will handle write locking etc. For my information, how is LVS going to help here? Is it increasing throughput to disk?.

Don Hinshaw

Would a simple TCP_CONNECT be the right way to test NFS?

Horms

If you are running NFS over TCP/IP then perhaps, but in my experience NFS deployments typically use NFS over UDP/IP. I'm suspecting a better test would be to issue some rpc calls to the NFS server and look at the response, if any. Something equivalent to what showmount can do might not be a bad start.

Joe

how about exporting the directory to the director as well and doing an `ls` every so often

Malcolm Cowe malcolm_cowe (at) agilent (dot) com 7 Aug 2001

HP's MC/ServiceGuard cluster software monitors NFS using the "rpcinfo" command -- it can be quite sensitive to network congestion, but it is probably the best tool for monitoring this kind of service.

The problem with listing an NFS exported directory is that when the service bombs, ls will wait for quite a while -- you don't want the director hanging because of an NFS query that's gone awry.

Nathan Patrick np (at) sonic (dot) net 09 Aug 2001

Mon includes a small program which extends what "rpcinfo" can do (and shares some code!) Look for rpc.monitor.c in the mon package, available from kernel.org among other places. In a nutshell, you can check all RPC services or specific RPC services on a remote host to make sure that they respond (via the RPC null procedure). This, of course, implies that the portmapper is up and responding.

From the README.rpc.monitor file:

This program is a monitor for RPC-based services such as the NFS protocol, NIS, and anything else that is based on the RPC protocol. Some general examples of RPC failures that this program can detect are:

- missing and malfunctioning RPC daemons (such as mountd and nfsd) - systems that are mostly down (responding to ping and maybe accepting TCP connections, but not much else is working) - systems that are extremely overloaded (and start timing out simple RPC requests)

To test services, the monitor queries the portmapper for a listing of RPC programs and then optionally tests programs using the RPC null procedure.

At Transmeta, we use:

"rpc.monitor -a" to monitor Network Appliance filers "rpc.monitor -r mountd -r nfs" to monitor Linux and Sun systems

Michael E Brown michael_e_brown (at) dell (dot) com 08 Aug 2001

how about rpcping?

(Joe - rpcping is at nocol_xxx.tar.gz for those of us who didn't know rpcping existed.)

Steven Lang Sep 26, 2001

The primary protocol I am interested in here is NFS. I have the director setup with DR with LC scheduling, no persistence, with UDP connections timing out after 5 seconds. I figured the time it would need to be accessing the same host would be when reading a file, so they are not all in contention for the same file, which seems to cost preformance in GFS. That would all come in a series of accesses. So there is not much need to keep the traffic to the same host beyond 5 seconds.

Horms horms (at) vergenet (dot) net

I know this isn't the problem you are asking about, but I think there are some problems with your architecture. I spent far to much of last month delving into NFS - for reasons not related to laad balancing - and here are some of the problems I see with your design. I hope they are useful.

As far as I can work out you'll need the persistance to be much longer than 5s. NFS is stateless, but regardless, when a client connects to a server a small ammount of state is established on both sides. The stateless aspect comes into play in that if either side times out, or a reboot is detected then the client will attempt to reconnect to the server. If the client doesn't reconnect, but rather its packets end up on a different server because of load balancing the server won't know anything about the client and nothing good will come of the sitiation. The solution to this is to ensure a client consistently talks to the same server, by setting a really long persistancy.

There is also the small issue of locks. lockd should be running on the server and keeping track of all the locks the client has. If the client has to reconnect, then it assumes all its locks are lost, but in the mean time it assumes everything is consistent. If it isn't accessing the same server (which wouldn't work for the reason given above) then the server won't know about any locks it things it has.

Of course unless you can get the lockd on the different NFS servers to talk to each other you are going to have a problem if different clients connected to different servers want to lock the same file. I think if you want to have any locking you're in trouble.

I actually specifically tested for this. Now it may just be a linux thing, not common to all NFS platforms, but in my tests, when packets were sent to the server other than the one mounted, it happily serves up the requested data without even blinking. So whatever state is being saved (And I do know there is some minimal state) seems unimportant. It actually surprised me how seamlessly it all worked, as I was expecting the non-mounted server to reject the requests or something similar.

That is quite suprising as the server should maintain state as to what clients have what mounted.

There is also the small issue of locks. lockd should be running on the server and keeping track of all the locks the client has. If the client has to reconnect, then it assumes all its locks are lost, but in the mean time it assumes everything is consistent. If it isn't accessing the same server (which wouldn't work for the reason given above) then the server won't know about any locks it things it has.

This could indeed be an issue. Perhaps setting up persistence for locks. But I don't think it is all bad. Of course, I am basing this off several assumptions that I have not verified at the moment. I am assuming that NFS and GFS will Do The Right Thing. I am assuming the NFS lock daemon will coordinate locks with the underlying OS. I am also assuming that GFS will then coordinate the locking with the rest of the servers in the cluster. Now as I understand locking on UNIX, it is only advisory and not enforced by the kernel, the programs are supposed to police themselves. So in that case, as long as it talks to a lock daemon, and keeps talking to the same lock daemon, it should be okay, even if the lock daemon is not on the server it is talking to, right?

That should be correct, the locks are advisor. As long as the lock daemons are talking to the underlying file system (GFS) then the behaviour should be correct, regarldess of which lock daemon a client talks to, as long as the client consistently talks to the same lock daemon for a given lock.

Of course, in the case of a failure, I don't know what will happen. I will definitely have to look at the whole locking thing in more detail to know if things work right or not, and maybe get the lock daemons to coordinate locking.

Given that lockd currently lives entirely in the kernel that is easier said than done.

This is from the beowulf mailing list

Ronald G Minnich rminnich (at) lanl (dot) gov 26 Sep 2002

NFS is dead for clusters. We are targeting three possible systems, each having a different set of advantages:

• panasas (http://www.panasas.com)

• lustre (http://www.lustre.org)

• v9fs (http://v9fs.sourceforge.net), from yours truly ^[7]

This is not LVS exactly, but here's from a discussion on hard and soft mounts from the beowulf mailing list.

Alvin Oga alvin (at) Maggie (dot) Linux-Consulting (dot) com 14 May 2003

if you export a raid subsystem via nfs... you will have problems

• if you export it w/ hard mount, all machines sit and wait for the machine that went down to come back up

• if you export it w/ soft mount, you can ^C out of any commands that try to access the machine went down, and continue on your merry way as long as you didnt need its data

best way to get around NFS export problems - have 2 or 3 redundant systems of identical data ( a cluster of data ... like that of www.foo.com websites )

Greg Lindahl lindahl (at) keyresearch (dot) com 15 May 2003

There are soft and hard mounts, and there are interruptable mounts ("intr" -- check out "man nfs").

• A hard mount will never time out. If you make it interruptable, then the user can choose to ^C. This is the safe option.

• A soft mount will time out, typically leaving you with data corruption for any file open at the time. This is the option you probably never want to use.

By the way, if you use the automounter to make sure that unused NFS filesystems are not mounted, it can help quite a bit when you have a crash, depending on your usage pattern.

Other potential problems include: inadequate power or airflow, high input air temperature, old BIOS on the 3ware card, old Linux kernel, etc.

Benjamin Lee ben (at) au (dot) atlas-associates (dot) com

I am wondering what (currently) people consider a production quality platform to run an NFS server on? I am thinking maybe a BSD although the kernel NFS code in Linux is much more stable now, I've heard. The idea is to put together a RAID boxen which will serve web pages and mail spool via NFS. (Don't worry, I'm only mounting the mail spool once. ;-) It's not a large enterprise system. )

Ted Pavlic tpavlic (at) netwalk (dot) com 19 Sep 2000

While Linux does not provide very good NFS support and generally has problems with things like locking (major problems), and quotas, don't rule it out. Personally I have a system now which is very similar to the system in which you sound like you want to build. Until recently, it was a Linux server with an external RAID serving both mailspool and web pages to four realservers. (and a couple of other servers -- application, news, etc.) Right now (for various reasons most having to do with how nightly backups are handled) I have two Linux servers both with external RAIDs. One handles mailspool, the other handles web pages. Both are configured so that if and when one machine Linux server dies completely the other can pick up the other RAID and serve both RAIDs again. There will be some manual interaction, of course, but I have NOCs 24/7 and hopefully such a problem would occur when a tech was available to handle the switch.

Such a system seems to work fine for me. (It's worked for a long time). It's easy for me to administrate because everything's Linux. I run into problems here or there when a program has trouble locking and requires a local share, but I get around those problems... and overall it's not that bad.

Joe 16 Dec 2005

The client gets a stale file handle when the client has a file||directory open and the server stops serving the file||directory. This error is part of the protocol (i.e. it's not a bug or a problem, it's supposed to happen). The stale file handle doesn't happen at the server end (the server doesn't care if the client is totally hosed).

client                     server

                           server:# export /home/user/
                           server:# ls /home/user
                                  foo

client:# mount /home/user
client:# cd /home/user/
client:# ls ./
          foo

client:# cd foo
client:# ls
    ..listing of files in foo

                          server:# unexport /home/user/

client:# ls
      error: stale file handle

df and mount will hang (or possibly return after a long timeout). The error goes away when the server comes back.

                          server#: export /home/user

client:# ls
    .. listing of files in foo

If the servers went down and the clients (realservers) stayed up, all with open file handles, the clients just have to wait till the servers come up again. The stale file handle will mess up the client till the server comes back. Since foo is on the same piece of disk real-estate, foo comes back with the same file handle when the server reappears

An irrecoverable example:

                            server:# export /home/user

client:# mount /home/user
client:# cd /home/user
client:# ls ./
    foo

(ie as before so far)

do something different, 
force an irreversible failure on the server

                            server:# rmdir /home/user/foo

client:# ls ./

    stale file handle

                            server:# mkdir /home/user/foo

client:# ls ./
    stale file handle

In this example, when /home/user/foo is recreated, it's on a new piece of disk real-estate and will have a different file handle. The client is hung and you can't umount /home/user (maybe you can with umount -f). If you can't umount /home/user, you will have to reboot the client.

Joseph L. Kaiser 7 Mar 2006

I have been tasked to mount a read-only NFS mounted software area to 500+ nodes. I wanted to do this with NFS and LVS, but after reading all the howtos with regard to NFS and LVS and reading all the email with regard to this in the archives (twice), it seems clear to me that this doesn't work.

Joe

I haven't done any of this, I've just talked to people - so my knowledge is only theoretical.

(ro) is a different kettle of fish. If you use identical disks (you only need identical geometry, so you could in principle use different model disks) and make the disks bitwise identical (e.g. made with dd), then clients will always get the same filehandle for the same file, no matter which realserver they connect to. For disk failure, make sure you have a few dd copies spare. I assume (but don't know) you won't be able to use RAID etc because of problems keeping the individual disks identical on the different realservers. You'll only be able to use single disks.

If you have to update files, then you'll have to do it bitwise. I don't have any ideas on how to do this off the top of my head. Perhaps you could have two disks, one mounted and the other umounted, but having received the new dd copy. You would change the weight to 0 with ipvsadm and let the connections expire (about 3 mins TCP UDP scheduling) umount the current disk and then mount the new disk. Presumably you'll have some period in the changeover where the realservers are presenting different files.

Presumably for speed, you should have as much memory in the realservers as possible, so that recently accessed files are still in memory.

However, I have a boss, and he wanted me to ask if turning off no-attribute caching (noac) would help in the reliability of this service.

If the disks are (ro) then the attributes on the server will never change, in which case you want the client to cache the attributes forever.

He has seen with another NFS mounted filesystem that using the "noac" turns off caching and clients that sometimes get "Stale NFS file handle" will reread the file and succeed.

Is having the client not reboot only "sometimes" an acceptable spec? With "noac" will it now pass the tests in stale file handle?

Neamtu Dan dlxneamtu(at)yahoo(dot)com

I have a test system made up by a director, 3 servers running ftp and http and 2 storage computers from where the servers take their data. If I were to stick to one storage computer there would be no problem, but I want redundacy. So I've set up a working heartbeat on both the storage computers, but when the primary fails the servers can no longer use the data unless the servers umount and mount again on the Virtual IP address the heartbeat uses (I tryed mounting with nfs and samba so far). As I understand these 2 mounting methods will not work in case of failover because of different disk geometry. Do you know what I should do for nfs or samba to work in case of failover (at least for read only, if read-write on the storage is not possible).

mike mike503 (at) gmail (dot) com 1 Apr 2006

drbd+NFS works for me, pretty well too. check out linux-ha.org for HaNFS.