My goal here is to describe the process of setting up a Monitoring system for a High Availability network security appliance. Specifically, this is work for a customer, who is going to implement one of our AGORA systems (see my earlier blog from this week) in a High Availability configuration. A specific feature of this monitoring system, is that it should detect failure of a primary system, and switch to a secondary automatically, according to a set of rules.
Now High-Availability means different things to different people. In my case, I interpret it to mean any system which when correctly implemented, will reduce the probability of a systems failure. As a system is made up from different parts, we isolate those subsystems which are most likely to fail, and put measures in place to detect or prevent this failure.
My goal here is to develop a network-based monitoring sub-system, which will continuously monitor and measure performance of the target system, and to activate special counter-measures in the event of a subsystem failure. I plan to use off-the-shelf components wherever possible, and especially open-source tools running in a Linux environment (although not all tools selected are of this type.) I believe this approach will be helpful to document, in case others want to adopt a similar approach, and you can learn from my mistakes.
First is to choose to development environment. I am going to develop within a VMWare appliance, which is a Virtual Machine. By doing this, it will be easier for the customer to implement at their own site. I happen to be using a MacBook Pro for this work, but it could easily be an Ubuntu Linux or even Windows XP box. Some of the features and tools I plan to implement include:
- Cacti -- used for time-series graphing of various metrics. In particular, useful for showing trends.
- SmokePing -- a nice Cacti-based tool to show network latency. Network performance is of particular interest.
- Perl -- the general purpose scripting language for writing new functionality
- XAMPP -- one of my favourite bundles of Apache, MySQL, Perl and PHP
- Mon/Nagios/Hobbit -- select from one of several network monitoring tools
- VMware -- used to run a virtual machine, for portability
- CentOS -- the version of Linux I chose for running the monitoring system inside the VMWare
I considered using a solution such as keepalived, but thought that might be more complex than I need. Plus I like re-inventing wheels...
Preparing the Development System
My first task is to connect to our VMWare server, and build the development environment. This box is stored in our data centre, and only provides access via SSH. Therefore, I am going to tunnel in via SSH, using VNC to get access to the graphical environment.
For the Mac OSX, I have chosen to use "Chicken of the VNC" as my VNC client. Because I need to tunnel in via SSH, I chose to open a terminal window, and type in the command directly.
ssh root@vmware-dev -L 5901:localhost:5901
I then connect to localhost port 1 in the VNC client, which will then tunnel to the remote system. Entering the password, and I am faced with the screen shot below.
Now I use the interface of the VMWare server, and tell it I want to create a virtual machine, using the Red Hat Enterprise Linux 4 template (which is closest to CentOS.) I choose also only 640 Mb of RAM (this machine will be running as a Web server, but I won't install X11.) I don't need a physical CD, as I have downloaded the ISO images of the CentOS onto the VMWare server, and just need to mount the image as if it was the CD drive. I switch on the VM, and it boots immediately into the CentOS installer.
I run through the installation options, selecting mostly the defaults. I made the VM with only 8 Gb of disk, so I have chosen a minimal install. I'll add the other stuff I need later. My first step however will be to use YUM to install any required security patches and updates for the minimal install, then download and install my Web environment, XAMPP. I will also add the VMWare tools, as these are important if I want the system to have good time synchronization (which is important for security applications), because NTP and friends don't play together nicely with Virtual Machines due to clock tick latency correction.
Here are the commands used:
wget http://www.apachefriends.org/download.php?xampp-linux-1.6.4.tar.gz
yum update
There was around 48 Mb of updates for the CentOS packages -- mostly new versions of tools and the kernel, with a few minor security issues.
See the Apache Friends web site for details on installing XAMPP. Just follow the instructions for improving its security, and make it run from startup by using chkconfig to add it to the processes to be run upon a reboot (after symbolic linking into /etc/init.d).
Smokeping
My first choice was to install Smokeping, by Tobias Oetiker. It's a great tool for visualization of network behaviour, which is an important part of any network-based services. I simply followed the comprehensive installation guide. Later, I found a more friendly Smokeping install guide here.
For convenience of the reader, I will paste below the commands needed. I decided to use binary distributions, rather than building from source, to save installing to many prerequisites in the VM.
yum install libart_lgpl
yum install perl-Time-HiRes
wget http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-1.2.23-1.el4.rf.i386.rpm
wget http://dag.wieers.com/rpm/packages/rrdtool/perl-rrdtool-1.2.23-1.el4.rf.i386.rpm
# Note both RPMs have to be installed with a single command, to avoid a dependency loop
rpm -Uvh rrdtool-1.2.23-1.el4.rf.i386.rpm perl-rrdtool-1.2.23-1.el4.rf.i386.rpm
wget http://downloads.sourceforge.net/echoping/echoping-6.0.2.tar.gz?use_mirror=heanet
yum install curl
I'll continue this in Part 2.
No comments:
Post a Comment