Version 7 (modified by work@…, 11 years ago) (diff) |
---|
Server Administration
This document describes how the current server installation is set up
Apache Tomcat
Currently, every environment has it's own dedicated Apache Tomcat application container where the production version has a memory limit set to 2GB and the others are set to 512MB. The separate containers allow easy restarting of certain environments without impacting the other environments. The following environments have currently been set up in /home/tomcat:
- apache-tomcat-www --> production environment
- apache-tomcat-nmcdsptest --> the test environment; note that this is not called 'test' but nmcdsptest. The reason is that Grails' built in testsuite uses the test environment and expects an hsqldb, while for test server environment we use PostgreSQL and numerous other tweaks.
- apache-tomcat-ci --> the continuous integration environment
In /home/tomcat also some other folders can be seen:
- apache-tomcat-tools --> the tomcat instance running specific tools (like nexus and hudson)
- apache-tomcat-dbnptest --> a fake tomcat folder, used for building the dbnptest package
- apache-tomcat-dbnpdemo --> alias to dbnptest, used to building the dbnpdemo package
Scripts for managing tomcat
In /home/tomcat/scripts a couple of scripts are available for easy handling the tomcat instances:
- kill_tomcat.sh --> kills a running tomcat instance
- restart_tomcat.sh --> restarts a running tomcat instance, however kill_tomcat.sh is prefered here as check_tomcat.sh would take care of starting tomcat with the proper settings
- check_tomcat.sh --> checks if a given tomcat is running, and starts it if it is not running
Some of these scripts are currently executed using a cronjob:
tomcat@nmcdsp ~/scripts $ crontab -l # m h dom mon dow command ... # Check if tomcat is running properly * * * * * /home/tomcat/scripts/check_tomcat.sh www >> /home/tomcat/logs/check_tomcat.log 2>&1 * * * * * /home/tomcat/scripts/check_tomcat.sh ci >> /home/tomcat/logs/check_tomcat.log 2>&1 * * * * * /home/tomcat/scripts/check_tomcat.sh nmcdsptest >> /home/tomcat/logs/check_tomcat.log 2>&1 * * * * * /home/tomcat/scripts/check_tomcat.sh tools >> /home/tomcat/logs/check_tomcat.log 2>&1 # restart tomcat automatically every day 0 6 * * * /home/tomcat/scripts/kill_tomcat.sh www >> /home/tomcat/logs/restart_tomcat.log 2>&1 0 6 * * * /home/tomcat/scripts/kill_tomcat.sh ci >> /home/tomcat/logs/restart_tomcat.log 2>&1 0 6 * * * /home/tomcat/scripts/kill_tomcat.sh nmcdsptest >> /home/tomcat/logs/restart_tomcat.log 2>&1 0 6 * * * /home/tomcat/scripts/kill_tomcat.sh tools >> /home/tomcat/logs/restart_tomcat.log 2>&1 # and kill ci every 3 hours 0 */2 * * * /home/tomcat/scripts/kill_tomcat.sh ci >> /home/tomcat/logs/restart_tomcat.log 2>&1 ... tomcat@nmcdsp ~/scripts $
If, for some reason, a tomcat needs to be restarted, it's best to use these scripts for doing so as they will also make sure tomcat is started with the proper memory settings. So, if you for example want to restart the ci instance, you would execute:
tomcat@nmcdsp ~ $ ~/scripts/kill_tomcat.sh ci 201106061658 killing tomcat ci instance (pid: 17992) tomcat@nmcdsp ~ $
The cronjob check_tomcat.sh ci will take care of starting the instance with the proper settings
Apache
Currently, the Apache webserver is set
Backups
Currently the backup cycle consists of two parts:
- database backups
- file upload backups
Database backups
The databases are backed up by a cronjob which runs as user posgres which currently dumps the production databases twice a day:
root@nmcdsp:~# su - postgres postgres@nmcdsp:~$ crontab -l # m h dom mon dow command 0 13,01 * * * ~/scripts/backup_database.sh gscf-www 2>&1 0 13,01 * * * ~/scripts/backup_database.sh nmcdsp-www 2>&1 0 13,01 * * * ~/scripts/backup_database.sh sam-www 2>&1 postgres@nmcdsp:~$
The database dumps are remotely stored on nbx14 in /home/nmcbackups/backups using the scponly nmcbackups account on nbx14. On nbx14 the backups folder is cleaned through another cronjob which runs as user root and keeps at least 5 backups files per environment and additionally keeps all backups that are less than 7 days old. This means that if there are 15 backup files for one product / environment (e.g. gscf-www) that are younger than 7 days, and two that are older than 7 days, only the latter two will be deleted.
root@nbx14:/root# crontab -l # The NMCDSP.org server has an automated backup process going, storing # database dumps in the (scponly) useraccount nmcbackup. This script # 1. cleans up the backup folder # 2. makes sure a minimal number of backups (per product, per environment) # are kept # 3. makes sure backups of a age less than a max age are kept 30 1,13 * * * /root/scripts/cleanup_backups >> /root/scripts/cleanup_backups.log 2>&1 root@nbx14:/root#
File backups
to be implemented...