Many people view serial ports as antiquated, out-dated connectors taking up space on their computers. However, serial ports still offer one of the best ways to communicate reliably and simply with a machine. For example, a serial port can be configured to act as a full-featured system console. This article describes how you can use cereal to monitor, log, and control access to serial lines connected to the consoles of other computers.
A computer today has several subsystems that should be run over the serial port to make it a full-featured system console:
Using a null-modem cable, you can connect the primary serial port on one computer to a remote computer's spare serial port. The remote computer can have full access to the first machine's console using only a terminal emulator like GNU screen or minicom. Not only do you not need yet another cumbersome monitor and keyboard, but you get a digital text stream to manipulate directly. You can log, timestamp, copy, paste, diff, etc the entire data stream flowing over the console, including boot-time spew, kernel crash output, etc.
Using serial ports as system consoles is especially useful in
high-density computing environments, such as racks in data centers.
Clutter like monitors, vga cables, video cards, keyboards, keyboard
cables, and kvm switches can all be thrown out in favor of serial
cables and a single computer that we will refer to as a serial
console server. A serial console server is a machine with many serial ports,
each connected to the serial console of another machine. An
administrator accesses the serial console server over the network
(e.g. via ssh) and from there can connect to the console of any of the
connected machines. In all examples below, scs
is the
hostname of a serial console server.
This is where cereal comes in. cereal is a management framework designed to monitor, log, and control access to serial lines. It is installed on the serial console server, and makes the task of maintaining such a machine simpler. It can also be used in simpler configurations, such as two machines which each monitor the other's console. This article describes how to setup and use cereal. We assume that the computers you are connecting to are already configured to send their consoles over the serial line. If that's not the case yet, read The Remote Serial Console HOWTO and get 'em configured first!
The cereal utility is not yet available in Debian proper, but it is packaged and available from the CMRG APT repository. Once you've added this repository to your sources.list (you may want to update your apt keys if you care to about verifying packages from the repo), you can pull it with apt-get or aptitude:
scs:0:~# echo deb http://cmrg.fifthhorseman.net/debian unstable cereal >> /etc/apt/sources.list.d/cereal.list scs:0:~# aptitude update scs:0:~# aptitude install cereal
Serial line management with cereal is broken out into sessions. A cereal session is a single runit service controlling a special screen session attached to a single serial line. Creation and control of these sessions is done with the cereal-admin utility:
scs:0:~# cereal-admin help Usage: cereal-admin[options] [args] Cereal session management program. subcommands: create (c) SESSION TTY BAUD USER LOGGROUP create cereal session start (s) [options] SESSION [SESSION]... start cereal session(s) -a (--all) start all sessions stop (k) [options] SESSION [SESSION]... stop cereal session(s) -a (--all) stop all sessions restart (r) [options] SESSION [SESSION]... restart cereal session(s) -a (--all) restart all sessions -r (--running) restart any currently running sessions destroy (d) [options] SESSION [SESSION]... destroy cereal session(s) -a (--all) destroy all sessions list (l) [SESSION]... list session(s) help (h,?) this help scs:0:~#
Let's assume the first computer whose serial console you want to
monitor with cereal is called lemur
. Let's assume
lemur's serial console is configured with baud rate 115200,
and is connected to the serial console server on /dev/ttyS1
(/dev/ttyS0 on scs is of course reserved for the
serial console server's own serial console!).
You can give access to cereal sessions, and the ability to read the session logs, to existing users. However, it is often a good idea to create new users and groups for these purposes. We begin by creating a new group that will be allowed to view ('follow') lemur's serial console logs, followed by creating a new user that will be able to control ('attach' to) lemur's cereal session.
scs:0:~# addgroup lemurviewers Adding group `lemurviewers' (GID 1001) ... Done. scs:0:~# adduser --ingroup lemurviewers --gecos 'lemur cereal session admin,,,' lemuradmin Adding user `lemuradmin' ... Adding new user `lemuradmin' (1001) with group `lemurviewers' ... Creating home directory `/home/lemuradmin' ... Copying files from `/etc/skel' ... scs:0:~#
You are now ready to create lemur's cereal session with cereal-admin:
scs:0:~# cereal-admin create lemur /dev/ttyS1 115200 lemuradmin lemurviewers Created session 'lemur': --f lemur /dev/ttyS1 115200 lemuradmin lemurviewers scs:0:~#
The first three characters of the session status line are the session flags that indicate the state of the cereal session. In order, they are:
Once a cereal session is created, you can begin monitoring
of it with the cereal-admin start
command:
scs:0:~# cereal-admin list lemur --f lemur /dev/ttyS1 115200 lemuradmin lemurviewers scs:0:~# cereal-admin start lemur Started session 'lemur'. scs:0:~# cereal-admin list lemur +-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers scs:0:~#
Now that you have a cereal session up and running, you can test the connection. Log into scs as the non-privileged lemuradmin user and use the cereal tool to attach to or follow the session:
lemuradmin@scs:~$ cereal help Usage: cereal[options] [args] Cereal client program. subcommands: attach (a) SESSION attach to session follow (f) [options] SESSION follow session log -c (--cat) cat log instead of follow -p (--path) output just log file path list (l) [SESSION]... list session(s) help (h,?) this help lemuradmin@scs:~$ cereal list lemur +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers lemuradmin@scs:~$
Using the attach function will put the the user into a cereal console, provided by a command-restricted screen session. To detach from an attached session, use "C-\ d". For help on other available commands, use "C-\ ?".
lemuradmin@scs:~$ cereal attach lemur
Debian GNU/Linux 4.0 lemur ttyS0 lemur login: root Password: Last login: Sat Oct 6 10:16:26 2007 on ttyS0 Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. lemur:0:~# exit cereal console: lemur 2007-10-06 10:17 "C-\ d" to detach
lemuradmin@scs:~$ cereal attach lemur [detached] lemuradmin@scs:~$
You can use the follow command to easily view the logs of a cereal session.
lemuradmin@scs:~$ cereal follow --cat lemur | cat -A ############################################################ ### cereal log: lemur ### +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers ### log file: /var/lib/cereal/sessions/lemur/log/main/current ############################################################ 2007-10-06_14:39:44.56895 2007-10-06_14:39:44.56900 cereal: session 'lemur' started. 2007-10-06_14:39:44.56900 starting screen session... 2007-10-06_14:39:58.86707 2007-10-06_14:39:58.86708 cereal: user 'lemuradmin' attaching to session... 2007-10-06_14:40:02.57346 2007-10-06_14:40:02.57348 Debian GNU/Linux 4.0 lemur ttyS0 2007-10-06_14:40:02.57348 2007-10-06_14:40:02.57349 lemur login: root 2007-10-06_14:40:13.58131 Password: 2007-10-06_14:40:14.58129 Last login: Sat Oct 6 10:16:48 2007 on ttyS0 2007-10-06_14:40:14.58129 Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 2007-10-06_14:40:14.58130 2007-10-06_14:40:14.58131 The programs included with the Debian GNU/Linux system are free software; 2007-10-06_14:40:14.58131 the exact distribution terms for each program are described in the 2007-10-06_14:40:14.58132 individual files in /usr/share/doc/*/copyright. 2007-10-06_14:40:14.58132 2007-10-06_14:40:14.58133 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent 2007-10-06_14:40:14.58134 permitted by applicable law. 2007-10-06_14:40:14.58135 Last was Sat 06 Oct 2007 10:33:15 AM EDT on ttyS0. 2007-10-06_14:40:14.58135 ESC[31;1mlemur:0:~#ESC[0m exit 2007-10-06_14:40:17.58525 logout 2007-10-06_14:40:17.58525 ESC[HESC[J 2007-10-06_14:40:18.58523 Debian GNU/Linux 4.0 lemur ttyS0 2007-10-06_14:40:18.58523 2007-10-06_14:40:18.58524 lemur login: 2007-10-06_14:40:19.62101 cereal: user 'lemuradmin' detached from session. lemuradmin@scs:~$
Now that you've created your first cereal session, you're ready to create serial sessions for all the rest of the attached serial consoles:
scs:0:~# cereal-admin list +-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers +-f chimp /dev/ttyS2 115200 chimpadmin chimpviewers +-f douc /dev/ttyS3 115200 doucadmin doucviewers +-f galago /dev/ttyS4 115200 galagoadmin galagoviewers +-f baboon /dev/ttyS5 115200 baboonadmin baboonviewers ... scs:0:~#
Cereal is developed and maintained by CMRG. Hopefully the package will be included in Debian soon, at which point bug tracking will be handled in the Debian BTS. Until then, please send all bug reports, questions, and suggestions to cereal AT fifthhorseman.net.