Difference between revisions of "Backup scripts"

From OpenKM Documentation
Jump to: navigation, search
 
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{TOCright}} __TOC__
 
{{TOCright}} __TOC__
  
−
These backup scripts use rsync to minimize network load and creates incremental backups, preserving last four backups. For more info, read http://www.mikerubel.org/computers/rsync_snapshots/.
+
These backup scripts use rsync to minimize network load and creates incremental backups, preserving last four backups. For more info, read http://www.mikerubel.org/computers/rsync_snapshots/. Also recommend the article [http://dar.linux.free.fr/doc/mini-howto/dar-differential-backup-mini-howto.es.html DAR differential backup mini-howto].
  
 
To install the cron job, run:
 
To install the cron job, run:
Line 23: Line 23:
 
{{Note|MAILTO may also be used to direct mail to multiple recipients by separating recipient users with a comma.}}
 
{{Note|MAILTO may also be used to direct mail to multiple recipients by separating recipient users with a comma.}}
  
−
For more information take a look at [http://adminschoice.com/crontab-quick-reference Crontab quick reference]
+
This script will export all databases from a MySQL installation:
  
−
== Remote backup with rsync ==
 
 
<source lang="bash">
 
<source lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#
 
#
 
## BEGIN CONFIG ##
 
## BEGIN CONFIG ##
−
HOST=$(uname -n)
+
MYSQL_PASS=""
−
FILES="/home/openkm"
+
DATABASE_EXP="/home/openkm/db"
−
BACKUP_DIR="/mnt/backup"
 
 
## END CONFIG ##
 
## END CONFIG ##
−
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
  
−
# Stop JBoss
+
rm -rf $DATABASE_EXP
−
/etc/init.d/jboss stop
+
mkdir -p $DATABASE_EXP
−
while [ "$(ps -ef | grep java | grep jboss | wc -l)" -gt "0" ]; do
 
−
  sleep 5; echo ".";
 
−
done
 
  
−
# Copy to backup server
+
# Backup de MySQL
−
rsync -apzhR --stats --delete --exclude=*~ --delete-excluded $FILES backup@server:$BACKUP_DIR/$HOST
+
if [ -n "$MYSQL_PASS" ]; then
−
 
+
  MYSQL_DBS=$(mysqlshow -h localhost -u root -p$MYSQL_PASS | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) { print $2 }');
−
# Start JBoss
+
−
/etc/init.d/jboss start
+
  for DB in $MYSQL_DBS; do
−
echo -e "\n### END: $(date +"%x %X") ###"
+
    if [[ $DB != "mysql" && $DB != "test" && $DB != "information_schema" && $DB != "performance_schema" ]]; then
−
</source>
+
      echo "* Backuping MySQL data from $DB..."
−
 
+
      mysqldump -h localhost -u root -p$MYSQL_PASS $DB > $DATABASE_EXP/mysql_$DB.sql
−
== Remote backup with rsync and rotation ==
+
    fi
−
<source lang="bash">
+
  done
−
#!/bin/bash
+
fi
−
#
 
−
## BEGIN CONFIG ##
 
−
HOST=$(uname -n)
 
−
FILES="/home/openkm"
 
−
BACKUP_DIR="/mnt/backup"
 
−
## END CONFIG ##
 
−
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
−
 
 
−
# Stop JBoss
 
−
/etc/init.d/jboss stop
 
−
 
 
−
# Copy to backup server
 
−
ssh backup@server "cd $HOST; rm -rf backup.3; mv backup.2 backup.3; mv backup.1 backup.2; mv backup.0 backup.1"
 
−
rsync -apzhR --stats --delete --exclude=*~ --delete-excluded --link-dest="$BACKUP_DIR/$HOST/backup.1" \
 
−
$FILES backup@server:$BACKUP_DIR/$HOST/backup.0
 
−
 
 
−
# Start JBoss
 
−
/etc/init.d/jboss start
 
−
echo -e "\n### END: $(date +"%x %X") ###"
 
−
</source>
 
−
 
 
−
== Local backup with rsync and rotation ==
 
−
<source lang="bash">
 
−
#!/bin/bash
 
−
#
 
−
## BEGIN CONFIG ##
 
−
HOST=$(uname -n)
 
−
FILES="/home/openkm"
 
−
BACKUP_DIR="/mnt/backup"
 
−
## END CONFIG ##
 
−
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
−
 
 
−
# Stop JBoss
 
−
/etc/init.d/jboss stop
 
−
 
 
−
# Calculate snapshot
 
−
LAST_SNAPSHOT=`ls -ltr $BACKUP_DIR | tail -1 | awk {'print $8'} | cut -d . -f 2`
 
−
NEW_SNAPSHOT=$((LAST_SNAPSHOT+1))
 
−
 
 
−
# Copy to backup server
 
−
rsync -apzhR --stats --delete --exclude=*~ --exclude="$JBOSS_HOME/cache" --delete-excluded \
 
−
--link-dest="$BACKUP_DIR/$HOST/backup.$LAST_SNAPSHOT" $FILES "$BACKUP_DIR/$HOST/backup.$NEW_SNAPSHOT"
 
−
 
 
−
# Start JBoss
 
−
/etc/init.d/jboss start
 
−
echo -e "\n### END: $(date +"%x %X") ###"
 
−
 
 
−
# Status
 
−
echo "=================================";
 
−
du -hs $BACKUP_DIR
 
−
echo "*********************************";
 
−
du -hs --time $BACKUP_DIR/*
 
−
echo "=================================";
 
−
df -h | grep "$BACKUP_DIR"
 
−
echo "=================================";
 
−
</source>
 
−
 
 
−
== Remote backup with rdiff-backup ==
 
−
{{Note|rdiff-backup from Debian throws a warning due to a deprecated method [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=590107 os.popen2 is deprecated]], which can be hidden following the steps at [http://www.linuxinsight.com/how-to-shut-up-python-deprecation-warnings.html How to shut up Python deprecation warnings].}}
 
−
 
 
−
<source lang="bash">
 
−
#!/bin/bash
 
−
#
 
−
## BEGIN CONFIG ##
 
−
HOST=$(uname -n)
 
−
FILES="/home/openkm"
 
−
BACKUP_DIR="/mnt/backup"
 
−
## END CONFIG ##
 
−
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
−
 
 
−
# Stop JBoss
 
−
/etc/init.d/jboss stop
 
−
 
 
−
# Backup and purge old backups
 
−
rdiff-backup --remove-older-than 30B backup@server::$BACKUP_DIR/$HOST
 
−
rdiff-backup -v 3 --print-statistics --exclude /media --exclude /mnt \
 
−
  --include $FILES --exclude '**' / backup@server::$BACKUP_DIR/$HOST
 
−
 
 
−
# Start JBoss
 
−
/etc/init.d/jboss start
 
−
echo -e "\n### END: $(date +"%x %X") ###"
 
−
 
 
−
# Status
 
−
echo "=================================";
 
−
rdiff-backup --list-increment-sizes backup@server::$BACKUP_DIR/$HOST
 
−
echo "=================================";
 
 
</source>
 
</source>
  
−
More info about rdiff-backup:
+
This script will export all databases from a PostgreSQL installation:
−
* [http://rdiff-backup.nongnu.org/ rdiff-backup home page]
 
−
* [http://wiki.rdiff-backup.org/wiki/index.php/Main_Page rdiff-backup wiki]
 
−
 
 
−
== Local backup to USB disk with rdiff-backup ==
 
−
USB disk mount point can be defined in ''/etc/fstab'' as:
 
−
 
 
−
  /dev/sdb1      /mnt/backup    ext4    defaults        0      0
 
  
 
<source lang="bash">
 
<source lang="bash">
Line 154: Line 55:
 
#
 
#
 
## BEGIN CONFIG ##
 
## BEGIN CONFIG ##
−
HOST=$(uname -n)
+
DATABASE_EXP="/home/openkm/db"
−
MYSQL_PASS=""
 
−
OPENKM_HOME="/home/openkm"
 
−
JBOSS_HOME="$OPENKM_HOME/jboss-4.2.3.GA"
 
−
DATABASE_EXP="$OPENKM_HOME/db"
 
−
BACKUP_DIR="/mnt/backup"
 
 
## END CONFIG ##
 
## END CONFIG ##
  
−
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi
 
−
 
−
echo -e "### BEGIN: $(date +"%x %X") ###\n"
 
 
rm -rf $DATABASE_EXP
 
rm -rf $DATABASE_EXP
 
mkdir -p $DATABASE_EXP
 
mkdir -p $DATABASE_EXP
  
−
# Mount disk
+
# Backup de PostgreSQL
−
if mount | grep "$BACKUP_DIR type" > /dev/null; then
+
POSTGRESQL_DBS=$(su postgres -c "psql -l" | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) { print $1 }');
−
  echo "$BACKUP_DIR already mounted";
 
−
else
 
−
  mount $BACKUP_DIR;
 
−
fi
 
−
 
 
−
# Stop JBoss
 
−
/etc/init.d/jboss stop
 
−
 
 
−
# Clean logs
 
−
echo "Clean JBoss temporal files."
 
−
rm -rf $JBOSS_HOME/server/default/log
 
−
rm -rf $JBOSS_HOME/server/default/tmp
 
−
rm -rf $JBOSS_HOME/server/default/work
 
−
 
 
−
# Backup de MySQL
 
−
if [ -n "$MYSQL_PASS" ]; then
 
−
  MYSQL_DBS=$(mysqlshow -h localhost -u root -p$MYSQL_PASS | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) && ($2 != "mysql") && ($2 != "test") && ($2 != "information_schema") { print $2 }');
 
−
 
 
−
  for DB in $MYSQL_DBS ; do
 
−
    echo "* Backuping MySQL data from $DB...";
 
−
    mysqldump -h localhost -u root -p$MYSQL_PASS $DB > $DATABASE_EXP/mysql_$DB.sql
 
−
  done
 
−
  echo "-------------------------------------";
 
−
fi
 
−
 
 
−
# Backup and purge old backups
 
−
rdiff-backup --remove-older-than 30B $BACKUP_DIR/$HOST
 
−
rdiff-backup -v 3 --print-statistics --exclude /media --exclude /mnt \
 
−
  --include $OPENKM_HOME --exclude '**' / $BACKUP_DIR/$HOST
 
  
−
# Start JBoss
+
for DB in $POSTGRESQL_DBS ; do
−
/etc/init.d/jboss start
+
  if [[ $DB != "template0" && $DB != "template1" && $DB != "postgres" ]]; then
−
echo -e "\n### END: $(date +"%x %X") ###"
+
    echo "* Backuping PostgreSQL data from $DB..."
−
 
+
    su postgres -c "pg_dump -Fc -b $DB" > $DATABASE_EXP/pg_$DB.dmp
−
# Status
+
   fi
−
echo "=================================";
 
−
rdiff-backup --list-increment-sizes $BACKUP_DIR/$HOST
 
−
echo "*********************************";
 
−
df -h | grep "$BACKUP_DIR"
 
−
echo "=================================";
 
−
 
 
−
# Umount disk
 
−
sync
 
−
umount $BACKUP_DIR
 
−
</source>
 
−
 
 
−
Be sure to have the last JBoss startup script, otherwise use this piece of code to ensure that JBoss is stopped before backuping the files:
 
−
 
 
−
<source lang="bash">
 
−
while [ "$(ps -ef | grep java | grep jboss | wc -l)" -gt "0" ]; do
 
−
   sleep 5; echo ".";
 
 
done
 
done
 
</source>
 
</source>
  
−
== Backup with duplicity ==
+
For more information take a look at [http://adminschoice.com/crontab-quick-reference Crontab quick reference]
−
Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.
 
  
−
Duplicity can be installed in Debian / Ubuntu as simple as:
+
* [[Backup with LFTP]]
−
 
+
* [[Backup with rsync]]
−
  $ sudo aptitude install duplicity
+
* [[Backup with rdiff-backup]]
−
 
+
* [[Backup with duplicity]]
−
But it not in the CentOS / RedHat default repositories, so you need to install from another source. This script will help in this installation process:
 
−
 
 
−
<source lang="bash">
 
−
#!/bin/bash
 
−
#ARCH=i386
 
−
ARCH=x86_64
 
−
 
 
−
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/duplicity-0.6.14-1.el5.$ARCH.rpm
 
−
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/ncftp-3.2.2-1.el5.$ARCH.rpm
 
−
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/librsync-0.9.7-13.el5.$ARCH.rpm
 
−
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-GnuPGInterface-0.3.2-2.el5.noarch.rpm
 
−
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-boto-1.9b-6.el5.noarch.rpm
 
−
 
 
−
rpm -Uvh duplicity-0.6.14-1.el5.$ARCH.rpm ncftp-3.2.2-1.el5.$ARCH.rpm librsync-0.9.7-13.el5.$ARCH.rpm python-GnuPGInterface-0.3.2-2.el5.noarch.rpm python-boto-1.9b-6.el5.noarch.rpm
 
−
</source>
 
  
 
[[Category: Installation Guide]]
 
[[Category: Installation Guide]]
−
[[Category: OKM Network]]
 

Latest revision as of 15:01, 10 March 2014

These backup scripts use rsync to minimize network load and creates incremental backups, preserving last four backups. For more info, read http://www.mikerubel.org/computers/rsync_snapshots/. Also recommend the article DAR differential backup mini-howto.

To install the cron job, run: 

$ sudo crontab -e

And add these lines according to your personal configuration: 

MAILTO=nomail@openkm.com
@weekly /root/backup.sh | tee -a /root/backup.log

Or, if you want to separate log reports by date: 

MAILTO=nomail@openkm.com
@weekly /root/backup.sh | tee /root/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log

Nota clasica.png MAILTO may also be used to direct mail to multiple recipients by separating recipient users with a comma.

This script will export all databases from a MySQL installation: 

#!/bin/bash
#
## BEGIN CONFIG ##
MYSQL_PASS=""
DATABASE_EXP="/home/openkm/db"
## END CONFIG ##

rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP

# Backup de MySQL
if [ -n "$MYSQL_PASS" ]; then
  MYSQL_DBS=$(mysqlshow -h localhost -u root -p$MYSQL_PASS | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) { print $2 }');
 
  for DB in $MYSQL_DBS; do
    if [[ $DB != "mysql" && $DB != "test" && $DB != "information_schema" && $DB != "performance_schema" ]]; then
      echo "* Backuping MySQL data from $DB..."
      mysqldump -h localhost -u root -p$MYSQL_PASS $DB > $DATABASE_EXP/mysql_$DB.sql
    fi
  done
fi

This script will export all databases from a PostgreSQL installation: 

#!/bin/bash
#
## BEGIN CONFIG ##
DATABASE_EXP="/home/openkm/db"
## END CONFIG ##

rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP

# Backup de PostgreSQL
POSTGRESQL_DBS=$(su postgres -c "psql -l" | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) { print $1 }');

for DB in $POSTGRESQL_DBS ; do
  if [[ $DB != "template0" && $DB != "template1" && $DB != "postgres" ]]; then
    echo "* Backuping PostgreSQL data from $DB..."
    su postgres -c "pg_dump -Fc -b $DB" > $DATABASE_EXP/pg_$DB.dmp
  fi
done

For more information take a look at Crontab quick reference

Retrieved from "https://www.openkm.com/wiki/index.php?title=Backup_scripts&oldid=11336"