Server Administration

This document describes how the current server installation is set up

Apache Tomcat

Currently, every environment has it's own dedicated Apache Tomcat application container where the production version has a memory limit set to 2GB and the others are set to 512MB. The separate containers allow easy restarting of certain environments without impacting the other environments. The following environments have currently been set up in /home/tomcat:

• apache-tomcat-www --> production environment
• apache-tomcat-nmcdsptest --> the test environment; note that this is not called 'test' but nmcdsptest. The reason is that Grails' built in testsuite uses the test environment and expects an hsqldb, while for test server environment we use PostgreSQL and numerous other tweaks.
• apache-tomcat-ci --> the continuous integration environment

In /home/tomcat also some other folders can be seen:

• apache-tomcat-tools --> the tomcat instance running specific tools (like nexus and hudson)
• apache-tomcat-dbnptest --> a fake tomcat folder, used for building the dbnptest package
• apache-tomcat-dbnpdemo --> alias to dbnptest, used to building the dbnpdemo package

Scripts for managing tomcat

In /home/tomcat/scripts a couple of scripts are available for easy handling the tomcat instances:

• kill_tomcat.sh --> kills a running tomcat instance
• restart_tomcat.sh --> restarts a running tomcat instance, however kill_tomcat.sh is prefered here as check_tomcat.sh would take care of starting tomcat with the proper settings
• check_tomcat.sh --> checks if a given tomcat is running, and starts it if it is not running

Some of these scripts are currently executed using a cronjob:

tomcat@nmcdsp ~/scripts $ crontab -l
# m h  dom mon dow   command
...
# Check if tomcat is running properly
* * * * * /home/tomcat/scripts/check_tomcat.sh www >> /home/tomcat/logs/check_tomcat.log 2>&1
* * * * * /home/tomcat/scripts/check_tomcat.sh ci >> /home/tomcat/logs/check_tomcat.log 2>&1
* * * * * /home/tomcat/scripts/check_tomcat.sh nmcdsptest >> /home/tomcat/logs/check_tomcat.log 2>&1
* * * * * /home/tomcat/scripts/check_tomcat.sh tools >> /home/tomcat/logs/check_tomcat.log 2>&1

# restart tomcat automatically every day
0 6 * * * /home/tomcat/scripts/kill_tomcat.sh www >> /home/tomcat/logs/restart_tomcat.log 2>&1
0 6 * * * /home/tomcat/scripts/kill_tomcat.sh ci >> /home/tomcat/logs/restart_tomcat.log 2>&1
0 6 * * * /home/tomcat/scripts/kill_tomcat.sh nmcdsptest >> /home/tomcat/logs/restart_tomcat.log 2>&1
0 6 * * * /home/tomcat/scripts/kill_tomcat.sh tools >> /home/tomcat/logs/restart_tomcat.log 2>&1

# and kill ci every 3 hours
0 */2 * * * /home/tomcat/scripts/kill_tomcat.sh ci >> /home/tomcat/logs/restart_tomcat.log 2>&1
...
tomcat@nmcdsp ~/scripts $ 

If, for some reason, a tomcat needs to be restarted, it's best to use these scripts for doing so as they will also make sure tomcat is started with the proper memory settings. So, if you for example want to restart the ci instance, you would execute:

tomcat@nmcdsp ~ $ ~/scripts/kill_tomcat.sh ci
201106061658 killing tomcat ci instance (pid: 17992)
tomcat@nmcdsp ~ $ 

The cronjob check_tomcat.sh ci will take care of starting the instance with the proper settings

Apache

Currently, the Apache webserver is set

Backups

Currently the backup cycle consists of two parts:

• database backups
• file upload backups

Database backups

The databases are backed up by a cronjob which runs as user posgres which currently dumps the production databases twice a day:

root@nmcdsp:~# su - postgres
postgres@nmcdsp:~$ crontab -l
# m h  dom mon dow   command
0 13,01 * * * ~/scripts/backup_database.sh gscf-www 2>&1
0 13,01 * * * ~/scripts/backup_database.sh nmcdsp-www 2>&1
0 13,01 * * * ~/scripts/backup_database.sh sam-www 2>&1
postgres@nmcdsp:~$ 

The database dumps are remotely stored on nbx14 in /home/nmcbackups/backups using the scponly nmcbackups account on nbx14. On nbx14 the backups folder is cleaned through another cronjob which runs as user root and keeps at least 5 backups files per environment and additionally keeps all backups that are less than 7 days old. This means that if there are 15 backup files for one product / environment (e.g. gscf-www) that are younger than 7 days, and two that are older than 7 days, only the latter two will be deleted.

root@nbx14:/root# crontab -l
# The NMCDSP.org server has an automated backup process going, storing
# database dumps in the (scponly) useraccount nmcbackup. This script
#	1. cleans up the backup folder
#	2. makes sure a minimal number of backups (per product, per environment)
#	   are kept
#	3. makes sure backups of a age less than a max age are kept
30 1,13 * * * /root/scripts/cleanup_backups >> /root/scripts/cleanup_backups.log 2>&1
root@nbx14:/root# 

File backups

to be implemented...