Difference between revisions of "Backup scripts"

From OpenKM Documentation
Jump to: navigation, search
(Splitted in several articles)
Line 25: Line 25:
 
For more information take a look at [http://adminschoice.com/crontab-quick-reference Crontab quick reference]
 
For more information take a look at [http://adminschoice.com/crontab-quick-reference Crontab quick reference]
  
== Remote backup with rsync ==
+
* [[Backup with rsync]] [[File:Padlock.gif]]
<source lang="bash">
+
* [[Backup with rdiff-backup]] [[File:Padlock.gif]]
#!/bin/bash
+
* [[Backup with duplicity]] [[File:Padlock.gif]]
#
 
## BEGIN CONFIG ##
 
HOST=$(uname -n)
 
FILES="/home/openkm"
 
BACKUP_DIR="/mnt/backup"
 
## END CONFIG ##
 
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
 
 
# Stop JBoss
 
/etc/init.d/jboss stop
 
while [ "$(ps -ef | grep java | grep jboss | wc -l)" -gt "0" ]; do
 
  sleep 5; echo ".";
 
done
 
 
 
# Copy to backup server
 
rsync -apzhR --stats --delete --exclude=*~ --delete-excluded $FILES backup@server:$BACKUP_DIR/$HOST
 
 
 
# Start JBoss
 
/etc/init.d/jboss start
 
echo -e "\n### END: $(date +"%x %X") ###"
 
</source>
 
 
 
== Remote backup with rsync and rotation ==
 
<source lang="bash">
 
#!/bin/bash
 
#
 
## BEGIN CONFIG ##
 
HOST=$(uname -n)
 
FILES="/home/openkm"
 
BACKUP_DIR="/mnt/backup"
 
## END CONFIG ##
 
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
 
 
# Stop JBoss
 
/etc/init.d/jboss stop
 
 
 
# Copy to backup server
 
ssh backup@server "cd $HOST; rm -rf backup.3; mv backup.2 backup.3; mv backup.1 backup.2; mv backup.0 backup.1"
 
rsync -apzhR --stats --delete --exclude=*~ --delete-excluded --link-dest="$BACKUP_DIR/$HOST/backup.1" \
 
$FILES backup@server:$BACKUP_DIR/$HOST/backup.0
 
 
 
# Start JBoss
 
/etc/init.d/jboss start
 
echo -e "\n### END: $(date +"%x %X") ###"
 
</source>
 
 
 
== Local backup with rsync and rotation ==
 
<source lang="bash">
 
#!/bin/bash
 
#
 
## BEGIN CONFIG ##
 
HOST=$(uname -n)
 
FILES="/home/openkm"
 
BACKUP_DIR="/mnt/backup"
 
## END CONFIG ##
 
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
 
 
# Stop JBoss
 
/etc/init.d/jboss stop
 
 
 
# Calculate snapshot
 
LAST_SNAPSHOT=`ls -ltr $BACKUP_DIR | tail -1 | awk {'print $8'} | cut -d . -f 2`
 
NEW_SNAPSHOT=$((LAST_SNAPSHOT+1))
 
 
 
# Copy to backup server
 
rsync -apzhR --stats --delete --exclude=*~ --exclude="$JBOSS_HOME/cache" --delete-excluded \
 
--link-dest="$BACKUP_DIR/$HOST/backup.$LAST_SNAPSHOT" $FILES "$BACKUP_DIR/$HOST/backup.$NEW_SNAPSHOT"
 
 
 
# Start JBoss
 
/etc/init.d/jboss start
 
echo -e "\n### END: $(date +"%x %X") ###"
 
 
 
# Status
 
echo "=================================";
 
du -hs $BACKUP_DIR
 
echo "*********************************";
 
du -hs --time $BACKUP_DIR/*
 
echo "=================================";
 
df -h | grep "$BACKUP_DIR"
 
echo "=================================";
 
</source>
 
 
 
== Remote backup with rdiff-backup ==
 
{{Note|rdiff-backup from Debian throws a warning due to a deprecated method [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=590107 os.popen2 is deprecated]], which can be hidden following the steps at [http://www.linuxinsight.com/how-to-shut-up-python-deprecation-warnings.html How to shut up Python deprecation warnings].}}
 
 
 
<source lang="bash">
 
#!/bin/bash
 
#
 
## BEGIN CONFIG ##
 
HOST=$(uname -n)
 
FILES="/home/openkm"
 
BACKUP_DIR="/mnt/backup"
 
## END CONFIG ##
 
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
 
 
# Stop JBoss
 
/etc/init.d/jboss stop
 
 
 
# Backup and purge old backups
 
rdiff-backup --remove-older-than 30B backup@server::$BACKUP_DIR/$HOST
 
rdiff-backup -v 3 --print-statistics --exclude /media --exclude /mnt \
 
  --include $FILES --exclude '**' / backup@server::$BACKUP_DIR/$HOST
 
 
 
# Start JBoss
 
/etc/init.d/jboss start
 
echo -e "\n### END: $(date +"%x %X") ###"
 
 
 
# Status
 
echo "=================================";
 
rdiff-backup --list-increment-sizes backup@server::$BACKUP_DIR/$HOST
 
echo "=================================";
 
</source>
 
 
 
More info about rdiff-backup:
 
* [http://rdiff-backup.nongnu.org/ rdiff-backup home page]
 
* [http://wiki.rdiff-backup.org/wiki/index.php/Main_Page rdiff-backup wiki]
 
 
 
== Local backup to USB disk with rdiff-backup ==
 
USB disk mount point can be defined in ''/etc/fstab'' as:
 
 
 
  /dev/sdb1      /mnt/backup    ext4    defaults        0      0
 
 
 
<source lang="bash">
 
#!/bin/bash
 
#
 
## BEGIN CONFIG ##
 
HOST=$(uname -n)
 
MYSQL_PASS=""
 
OPENKM_HOME="/home/openkm"
 
JBOSS_HOME="$OPENKM_HOME/jboss-4.2.3.GA"
 
DATABASE_EXP="$OPENKM_HOME/db"
 
BACKUP_DIR="/mnt/backup"
 
## END CONFIG ##
 
 
 
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi
 
 
 
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
rm -rf $DATABASE_EXP
 
mkdir -p $DATABASE_EXP
 
 
 
# Mount disk
 
if mount | grep "$BACKUP_DIR type" > /dev/null; then
 
  echo "$BACKUP_DIR already mounted";
 
else
 
  mount $BACKUP_DIR;
 
fi
 
 
 
# Stop JBoss
 
/etc/init.d/jboss stop
 
 
 
# Clean logs
 
echo "Clean JBoss temporal files."
 
rm -rf $JBOSS_HOME/server/default/log
 
rm -rf $JBOSS_HOME/server/default/tmp
 
rm -rf $JBOSS_HOME/server/default/work
 
 
 
# Backup de MySQL
 
if [ -n "$MYSQL_PASS" ]; then
 
  MYSQL_DBS=$(mysqlshow -h localhost -u root -p$MYSQL_PASS | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) && ($2 != "mysql") && ($2 != "test") && ($2 != "information_schema") { print $2 }');
 
 
 
  for DB in $MYSQL_DBS ; do
 
    echo "* Backuping MySQL data from $DB...";
 
    mysqldump -h localhost -u root -p$MYSQL_PASS $DB > $DATABASE_EXP/mysql_$DB.sql
 
  done
 
  echo "-------------------------------------";
 
fi
 
 
 
# Backup and purge old backups
 
rdiff-backup --remove-older-than 30B $BACKUP_DIR/$HOST
 
rdiff-backup -v 3 --print-statistics --exclude /media --exclude /mnt \
 
  --include $OPENKM_HOME --exclude '**' / $BACKUP_DIR/$HOST
 
 
 
# Start JBoss
 
/etc/init.d/jboss start
 
echo -e "\n### END: $(date +"%x %X") ###"
 
 
 
# Status
 
echo "=================================";
 
rdiff-backup --list-increment-sizes $BACKUP_DIR/$HOST
 
echo "*********************************";
 
df -h | grep "$BACKUP_DIR"
 
echo "=================================";
 
 
 
# Umount disk
 
sync
 
umount $BACKUP_DIR
 
</source>
 
 
 
Be sure to have the last JBoss startup script, otherwise use this piece of code to ensure that JBoss is stopped before backuping the files:
 
 
 
<source lang="bash">
 
while [ "$(ps -ef | grep java | grep jboss | wc -l)" -gt "0" ]; do
 
  sleep 5; echo ".";
 
done
 
</source>
 
 
 
== Backup with duplicity ==
 
Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.
 
 
 
Duplicity can be installed in Debian / Ubuntu as simple as:
 
 
 
  $ sudo aptitude install duplicity
 
 
 
But it not in the CentOS / RedHat default repositories, so you need to install from another source. This script will help in this installation process:
 
 
 
<source lang="bash">
 
#!/bin/bash
 
#ARCH=i386
 
ARCH=x86_64
 
 
 
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/duplicity-0.6.14-1.el5.$ARCH.rpm
 
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/ncftp-3.2.2-1.el5.$ARCH.rpm
 
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/librsync-0.9.7-13.el5.$ARCH.rpm
 
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-GnuPGInterface-0.3.2-2.el5.noarch.rpm
 
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-boto-1.9b-6.el5.noarch.rpm
 
 
 
rpm -Uvh duplicity-0.6.14-1.el5.$ARCH.rpm ncftp-3.2.2-1.el5.$ARCH.rpm librsync-0.9.7-13.el5.$ARCH.rpm python-GnuPGInterface-0.3.2-2.el5.noarch.rpm python-boto-1.9b-6.el5.noarch.rpm
 
</source>
 
  
 
[[Category: Installation Guide]]
 
[[Category: Installation Guide]]
 
[[Category: OKM Network]]
 
[[Category: OKM Network]]

Revision as of 12:29, 24 October 2011

These backup scripts use rsync to minimize network load and creates incremental backups, preserving last four backups. For more info, read http://www.mikerubel.org/computers/rsync_snapshots/.

To install the cron job, run:

$ sudo crontab -e

And add these lines according to your personal configuration:

MAILTO=nomail@openkm.com
@weekly /root/backup.sh | tee -a /root/backup.log

Or, if you want to separate log reports by date:

MAILTO=nomail@openkm.com
@weekly /root/backup.sh | tee /root/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log

Nota clasica.png MAILTO may also be used to direct mail to multiple recipients by separating recipient users with a comma.

For more information take a look at Crontab quick reference