Console Server

1 The Problem

You have a computer room with lots of UNIX® server machines and lots of communications hardware. Each of these machines needs a serial console. But serial terminals are hard to find and quite expensive (especially compared to a much more capable PC). And they take up a lot of precious space in the computer room.

You need access to the console because when things break, that is where error messages go. And some tasks have to be done on the console (e.g. boot problems or OS installs/upgrades). Some UNIX systems allow the console to break out to the ROM monitor which can sometimes be the only way to unstick a hung machine. This is often done with a LINE BREAK sent on the console serial port.

If we are going to play about with consoles, then there are a couple of other things that would be great:

• Remote access. Even in the same office, it would be convenient to access all the consoles from your desk without walking into the computer room. But often the machines are off-site, perhaps even in another country.

• Logging. If something has gone wrong, you would like to be able to have a look at the previous console output to see what is up. Ordinary console screens give you the last 25 lines. More would be better.

• Network Independence. The solution needs to work even if the network is down. After all, a failed network is when you need consoles the most! Even better is network independence with remote access.

• No single-point failure. A console system that crashes every machine when it fails is no use. This is particularly tricky with Sun UNIX hosts as they will interpret a powered-off terminal as a BREAK, and drop back to the ROM monitor.

• Interface with a pager or some similar alerter device.

• Ability to power-cycle machines remotely.

• Not be too expensive. Free is even better!

2 Possible Solutions

If you use PC hardware for your servers, then a so-called “KVM switch” is one possible solution. A KVM switch allows the use of a single keyboard, video screen and mouse for multiple boxes. This cuts down on the space problem, but only works for PC hardware (not any communications gear you might have), and is not accessible from outside the computer room. Nor does it have much scroll-back or logging, and you have to handle alerting some other way. The big downside is that it will not work for serial-only devices, such as communications hardware. This means that even with a room full of PC-based servers, you are probably still going to need some sort of serial console solution.

You might be tempted to do without a console terminal, but when things go pear-shaped you really need to see what is on the console. And you have to use the console to boot the machine and do things like OS upgrades or installs.

You might try having a single console terminal and switching from server to server as needed, either with a serial switch or just by patching it into the required machine. Serial switches are also hard to come by and not cheap, and may cause problems with sending BREAK when they switch. And (if your computer room is anything like ours) you never seem to have the right combination of patch leads to connect to the machine you need to, and even if the leads are there you can never work out exactly which combination of DTE/DCE headshells goes with which lead goes with which hardware. So you spend the first 10 minutes fooling around with breakout boxes and a box of leads, all while the server is down and the users are screaming. Of course this does not deal with the logging or remote access requirements. And inevitably the console is not switched to the machine you need so you lose all the console messages that might tell you what is going on.

One popular solution is to use terminal server hardware. Typically, the serial ports are connected to the various machine consoles, and set up for “reverse telnet” access. This means a user can telnet to a given IP/port and be connected to the appropriate console. This can be very cost-effective, as suitable old terminal servers can be picked up fairly cheaply (assuming you do not have a couple lying around). And it is of course network-accessible so suitable for remote access. But it suffers from one major drawback: if the network is down, then you have no access to any console, even if you are standing right next to the machine. (This may be partially alleviated by having a suitable terminal connected to one of the terminal server ports and connecting from there, but the terminal server software may not support that.) Also there is no logging or replay of console messages. But with a bit of work, and the addition of some software such as conserver (described below), this can be made to work pretty well.

A possibility suggested by Bron Gondwana is similar to the above solution. If you use servers with multiple serial ports, you can connect each spare serial port to the console port of the “next” server, creating a ring of console connections (in some sort of order). This can be made to work reasonably well with the aid of the conserver software, but can be a bit confusing otherwise (i.e. remembering which port is connected to which console). And you are stuck if you need to use serial ports for other things (such as modems) or you have machines without spare ports.

Or, if your budget exceeds your willingness to hack, you can buy an off-the-shelf solution. These vary in price and capability. See, for example, Lightwave, Perle, Avocent or Black Box. These solutions can be quite expensive - typically $USD100 - $USD400 per port.

3 Our Solution

In light of the above requirements, we chose a solution based on a dedicated PC running UNIX with a multiport serial card, and some software designed to handle serial consoles.

It includes the following elements:

• A surplus PC. We used a Pentium® 166, with a PCI bus, 2Gbyte hard disk and 64Mb of RAM. This is a massive overkill for this task, and P-100, 500Mb, 32Mb would be more than enough.

• A PC UNIX system. We used FreeBSD 4.3 as that is used for other tasks within our office.

• A multi-port serial card. We chose the EasyIO™ PCI 8-port card from Stallion Technologies. This cost us about $AUD740, or under $100/port, from Harris Technologies (which has lots of stuff but is by no means the cheapest place in town - shop around and you might get it a lot cheaper). This card has a big DB80 connector on the back, and a cable plugs into that which has a block with 8 RJ-45 sockets on it. (We chose the RJ-45 version as our entire cable plant is RJ-45. This allows us to patch connections from the required box to the console server without any special cables.) This is the only thing we needed to buy to make this all happen.

• We build two servers, one for each computer room, with 8 ports in one and 16 ports (via two EasyIO PCI cards) in the other. If we needed more than 16 ports, then another of the Stallion cards would be more cost-effective. We could conceivably support 128 ports in each server (with 2 EasyConnect 8/64 host cards and 8 16 port RJ-45 modules) for about $AUD12,000.

• A modem for remote access to the console server host when the network is down. We have not done this yet as the computer room is next door, but when we put a server in Sydney we will add the modem. The idea is that when the network is down, you can dial up and log into the server machine and run the console program locally. For security, we will probably leave the modem powered off and ask the gopher in Sydney to turn on the well-labelled button when we need it.

• A program called conserver. This program does all the magic required to enable remote access to consoles, and do the replaying and logging etc. It comes in two parts: a server called conserver that runs as a daemon and connects to the serial ports, handles logging etc, and a client program called console that can connect to the server, display console messages, send keystrokes (and BREAK), etc.

This design covers all the major requirements except remote power cycling:

• Remote access comes because the console client program works across the network.

• Logging is handled by the conserver program.

• If the network is down, then we can use the console on the PC to run the console client locally. For remote sites, we can add a modem for dial-in access to the server command line to run the client.

• By patching the Solaris™ servers (see Section 6), we can avoid pranging the whole computer room when the console server PC crashes (or the power supply fails, or whatever).

• We already have pager alerts from another system we have installed, but the console server has all the required log info so that could easily be implemented if we needed. And it even has a modem for calling the pager company!

• We do not currently support remote power cycling. Some versions of the conserver program support this, but it does require specialised serial-controlled power boards. We have no immediate need for remote power cycling (we have a gopher in each remote office who can do it by remote control) so this is not a major problem, and we could add it easily should we ever see the need and get the appropriate hardware.

• This solution was very cheap. Total cost for the 9-port server was $AUD750 for the IO card, as we re-used a surplus PC and already owned the hardware for the special cables. If we had to buy everything, then it would still only cost around $AUD1500 for the 8-port server.

4 Setting Up The Server

# cd /dev/ && sh MAKEDEV cuaE0

# make DEFAULTHOST=consolehost install

% ./configure --with-master=consoleserver --with-port=782

LOGDIR=/var/log/consoles
gallows:/dev/cuaE0:9600p:&:
roo:/dev/cuaE1:9600p:&:
kanga:/dev/cuaE2:9600p:&:
%%
allow:  itga.com.au
trusted:        127.0.0.1 buzz

@rands = ();
foreach (0..4) {
        push(@rands, rand 64);
}

$salt = join '', ('.', '/', 0..9, 'A'..'Z', 'a'..'z')[@rands];

$salt = '$1$' . $salt . '$';

print 'Enter password: ';
`stty -echo`;
$cleartext = <>;
`stty echo`;
chop($cleartext);
print crypt($cleartext, $salt), "\n";

% openssl passwd -1
Password: password
$1$VTd27V2G$eFu23iHpLvCBM5nQtNlKj/

cuaE0  "/usr/local/sbin/conserver" unknown on  insecure

#!/bin/sh
#
# Startup for conserver
#

PATH=/usr/bin:/usr/local/bin

case "$1" in
        'start')
                TTY=/dev/cuaE7
                conserver -d > $TTY
                # get NL->CR+NL mapping so msgs look right
                stty < /dev/cuaE7 opost onlcr
        echo -n ' conserver'
                ;;

        'stop')
        kill `cat /var/run/conserver.pid` && echo -n ' conserver'
                ;;

        *)
                echo "Usage: $0 { start | stop }"
                ;;

esac
exit 0

#
# The log files from conserver
/var/log/consoles/gallows   644  10    1000 *     Z /var/run/conserver.pid
/var/log/consoles/kanga     644  10    1000 *     Z /var/run/conserver.pid
/var/log/consoles/roo       644  10    1000 *     Z /var/run/conserver.pid

5 Cabling

This is always the hardest part of this kind of problem. We had only a dozen or so cables/headshells to build, and we already had a collection of the appropriate crimping tools and hardware, so we did it ourselves. But if you are not set up for this, or you have a large number of cables to make, then you might consider getting some cables custom made. Look in the yellow pages, there are a surprising number of places that do this! Getting custom-made cabling is good, and you can get much more professional results, but can be expensive. For example, the RJ-45 to DB-25 adapter kits described below are about $10 each; custom-made headshells are about twice that (and take a couple of weeks to arrive). Similarly, crimping custom RJ-45 to RJ-45 leads is quite cheap (say, $5 each) but it takes a fair amount of time. Custom made RJ-45 socket to RJ-45 plug converters cost about $25 each.

We have settled on RJ-45 Cat-V cabling for all our office and computer room cabling needs. This included patching between racks in the computer room. For serial connections, we use patchable headshells that have RJ-45 sockets on the back. This allows us to patch whatever RJ-45-DB-25 connections we need.

Which is just as well, because there are many incompatible ways to represent serial connections on the RJ-45 plug. So the cabling has to be very careful to use the right mapping.

6 On Sun Systems And Break

Anyone who has turned off a terminal used as a console for a Sun system will know what happens and why this is a problem. Sun hardware recognises a serial BREAK as a command to halt the OS and return to the ROM monitor prompt. A serial BREAK is an out-of-band signal on an RS-232 serial port that involves making the TX DATA line active (i.e. pulled down to less than -5V) for more than two whole character times (or about 2ms on a 9600bps line). Alas, this BREAK signal is all to easily generated by serial hardware during power-on or power-off. And the Stallion card does, in fact, generate breaks when the power to the PC fails. Unless fixed, this problem would mean that every Sun box connected to the console server would be halted whenever the power failed (due to dead power supplies, or fat-fingered operators unplugging it, or whatever). This is clearly not an acceptable situation.

Fortunately, Sun have come up with a set of fixes for this. For Solaris 2.6 and later, the kbd(1) command can be used to disable the ROM-on-BREAK behaviour. This is a good start, but leaves you out of luck in the situation where a break is needed to get into a broken machine.

Starting with Solaris 8, the kbd command can also be used to enable an alternate break sequence using the kbd -a alternate command. When this is set, the key sequence ReturnTildeCtrl-B (within 5 seconds) will drop to the ROM. You can enable this permanently by editing the /etc/default/kbd file; see the kbd(1) man page. Note that this alternate break sequence is only active once the kernel has started running multiuser and processed the default file. While the ROM is active (during power-on and during the boot process) and while running single-user, you still need to use a BREAK to get to the ROM prompt. The console client can cause the server to send a BREAK using the escape sequence Esccl1.

If you have a Sun software support contract, there are patches available for Solaris 2.6 and 2.7 that add the “alternate break” capability integrated into Solaris 2.8. Solaris 2.6 requires patch 105924-10 or higher. Solaris 2.7 requires patch 107589-02 or higher.

We have added this patch to all our Solaris 2.6 servers, and added it (and the entry in the /etc/default/kbd file) to our jumpstart configuration so it will automatically be added to every new install.

We have confirmed by direct testing that neither the Cisco 16xx, 26xx, or Catalyst hardware suffers from the BREAK sent when the Stallion card loses power. Contemporary Cisco software listens for BREAK signal only for first 30 seconds after power-on or reboot.

7 Using a Serial Console on FreeBSD

The procedure for doing this is described in detail in the FreeBSD Handbook. This is a quick summary.

ttyv1   "/usr/libexec/getty Pc"         cons25  on  secure

ttyd0   "/usr/libexec/getty std.9600"   dialup  off secure

ttyd0   "/usr/libexec/getty std.9600"   vt100   on secure

8 Security Implications

The client-server protocol for conserver requires the user of the console client to enter a password. This password is passed across the net in cleartext! This means conserver is not really suitable for use across untrusted networks (such as the Internet). Use of conserver-only passwords (in the conserver.passwd file) slightly mitigate this problem, but anyone sniffing a conserver connection can easily get console access, and from there prang your machine using the console break sequence. For operating across the Internet, use something secure like SSH to log into to the server machine, and run the console client there.

9 On Conserver Versions

The conserver program has fractured into a number of versions. The home page referenced below seems to be the latest and most featureful version around, and for July 2004 carries a version number of “8.1.9”. This is maintained by Bryan Stansell <[email protected]>, who has brought together the work of many people (listed on his webpage).

The FreeBSD ports collection contains a port for version 8.5 of conserver at comms/conserver. This seems to be older and less featureful than the 8.1.9 version (in particular, it does not support consoles connected to terminal server ports and does not support a conserver.passwd file), and is written in a fairly idiosyncratic manner (using a preprocessor to generate C code). Version 8.5 is maintained by Kevin S. Braunsdorf <[email protected]> who did most of the original work on conserver, and whose work Bryan Stansell is building on. The 8.5 version does support one feature not in the 8.1.9 version (controlling power to remote machines via a specific serial-interfaced power controller hardware).

Beginning with December 2001, Brian's version (currently 8.1.9) is also presented in ports collection at comms/conserver-com. We therefore recommend you to use this version as it is much more appropriate for console server building.

10 Links

11 Manual Pages

• console(8)

• conserver(8)

• conserver.cf(5)