[open-ils-commits] r835 - ESI-Examples/sys/scripts (miker)
svn at svn.open-ils.org
svn at svn.open-ils.org
Wed Mar 17 13:57:40 EDT 2010
Author: miker
Date: 2010-03-17 13:57:36 -0400 (Wed, 17 Mar 2010)
New Revision: 835
Added:
ESI-Examples/sys/scripts/README.db-backup
ESI-Examples/sys/scripts/eg-db-backup.sh
ESI-Examples/sys/scripts/eg-wal-archive.sh
Log:
adding db backup scripts, and an overview README of how to use them
Added: ESI-Examples/sys/scripts/README.db-backup
===================================================================
--- ESI-Examples/sys/scripts/README.db-backup (rev 0)
+++ ESI-Examples/sys/scripts/README.db-backup 2010-03-17 17:57:36 UTC (rev 835)
@@ -0,0 +1,67 @@
+README.db-backup
+----------------
+
+Evergreen uses the PostgreSQL database for it's backend data store. In order
+to correctly back up this database, specific steps must be taken. An rsync
+or tar/cpio archive of the data directory is not sufficient!
+
+For more information about why and how PostgreSQL backups work, see:
+
+http://www.postgresql.org/docs/8.3/static/continuous-archiving.html
+
+
+
+------------ Overview
+
+Included in this directory are two scripts:
+ * eg-db-backup.sh
+ * eg-wal-archive.sh
+
+These scripts facilitate the correct backup procedures for PostgreSQL in an
+Evergreen environment. Both must be edited to configure them for use in
+a new setup.
+
+
+
+------------- eg-db-backup.sh
+
+The eg-db-backup.sh script should be run on a regular basis, normally
+nightly, from the postgres user's crontab file. For example:
+
+0 6 * * * /home/postgres/eg-db-backup.sh
+
+
+This will create a base backup every morning at 6AM.
+
+
+
+------------- eg-wal-archive.sh
+
+The eg-wal-archive.sh is a PostgreSQL WAL archive_command-compatable script
+which will continuously archive the transaction files (WAL) that PostgreSQL
+generates as Evergreen transactions are processed. To use this script,
+set the archive_command variable in your production postgresql.conf file to:
+
+archive_command = '/home/postgres/eg-wal-archive.sh %p %f'
+
+(Assuming, of course, that /home/postgres/eg-wal-archive.sh is the path to the
+script on your system). Some versions of PostgreSQL will require a separate
+archive_mode setting be enabled in order to begin archiving WAL files.
+
+
+
+------------- Cleaning up
+
+PostgreSQL backups are relatively large, and you only need to keep the most
+recent base backup plus and WAL generated after the start of the backup in
+order to recover. To remove older backups, the following crontab entries are
+useful:
+
+
+0 5 * * * for i in `find /path/to/backup/directory/ -ctime +2`; do rm $i 2>/dev/null; done
+
+The path in the above 'find' command should match the directory into which you
+are archiving both base backups and WAL files. This should be added to the
+postgres user's crontab on the database server, and the crontab of the
+configured ARCHIVE_USER on any remote archive server.
+
Added: ESI-Examples/sys/scripts/eg-db-backup.sh
===================================================================
--- ESI-Examples/sys/scripts/eg-db-backup.sh (rev 0)
+++ ESI-Examples/sys/scripts/eg-db-backup.sh 2010-03-17 17:57:36 UTC (rev 835)
@@ -0,0 +1,108 @@
+#!/bin/bash
+# Evergreen database snapshot creation and archiving script
+# Copyright (C) 2008-2010 Equinox Software Inc.
+# Mike Rylander <mrylander at gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+#
+# You will need to edit the variables below to configure this script for
+# use in your production environment.
+#
+# This script must be run as the postgres user, and if snapshot shipping
+# is enabled then the postgres user should be able to log into the remote
+# host as the ARCHIVE_USER over ssh and scp with a passphraseless ssh key.
+#
+
+
+#--------------------- CONFIGURATION BEGIN ------------------------
+# Remove the following line once you have adjusted the configuration
+# below to match your environment.
+echo "Configuration not complete!" && exit 1;
+
+
+# Where the postgres binaries are installed, particularly psql
+PGBIN=/usr/local/bin/
+
+
+# Where the database cluster lives
+PGDATA=/usr/local/pgsql/data;
+
+
+# How to name the database snapshot files. Adjust to taste.
+ARCHIVE_LABEL=`date +MyEvergreen-production-postgres-backup-%FT%T`
+
+
+# Local snapshot archiving directory
+ARCHIVE_DIR="/var/backup/$HOSTNAME/evergreen/database/snapshot/"
+
+
+# Remote host (IP or resolvable name) to which snapshots should be shipped.
+# Leave empty to disable snapshot shipping.
+ARCHIVE_HOST=
+
+
+# User on the remote snapshot-receiving host.
+ARCHIVE_USER=
+
+# Snapshot archiving directory on the remote host, if snapshot shipping is
+# enabled.
+ARCHIVE_DST="/var/backup/$HOSTNAME/evergreen/database/snapshot/"
+
+#--------------------- CONFIGURATION END ------------------------
+
+
+
+
+ARCHIVE_FILE=$ARCHIVE_LABEL.cpio.gz
+# Make sure we're not overwriting an existing backup
+if [ -e $ARCHIVE_DIR/$ARCHIVE_FILE ]; then
+ echo "Cannot create backup: $ARCHIVE_DIR/$ARCHIVE_FILE exists";
+ exit;
+fi
+
+
+# Tell PG we're starting the backup
+START_RESULT=`$PGBIN/psql -tc "SELECT pg_start_backup('$ARCHIVE_LABEL') IS NOT NULL;"|grep t`
+if [ "_" == "_$START_RESULT" ]; then
+ echo "Could not start backup labeled $ARCHIVE_LABEL";
+ exit;
+fi
+
+
+# Grab the data we need (just copy it locally) ...
+(cd $PGDATA && find . -depth -print | grep -v pg_xlog | cpio -o | gzip > $ARCHIVE_DIR/$ARCHIVE_FILE)
+
+
+# ... tell PG we're done ...
+STOP_RESULT=`$PGBIN/psql -tc "SELECT pg_stop_backup() IS NOT NULL;"|grep t`
+if [ "_" == "_$STOP_RESULT" ]; then
+ echo "Could not stop backup labeled $ARCHIVE_LABEL";
+ exit;
+fi
+
+echo "Backup of database on $HOSTNAME complete. Archive label: $ARCHIVE_LABEL"
+
+if [ "_$ARCHIVE_HOST" != "_" ]; then
+ # ... then push it over to the backup host
+ scp -q $ARCHIVE_DIR/$ARCHIVE_FILE $ARCHIVE_USER@$ARCHIVE_HOST:$ARCHIVE_DST
+ SCP_RES=$?
+ if [ "$SCP_RES" != "0" ]; then
+ echo "Unable to archive $ARCHIVE_DIR/$ARCHIVE_FILE to $ARCHIVE_USER@$ARCHIVE_HOST:$ARCHIVE_DST!!"
+ exit;
+ fi
+ echo "Remote backup: $ARCHIVE_HOST:$ARCHIVE_DST"
+fi
+
Property changes on: ESI-Examples/sys/scripts/eg-db-backup.sh
___________________________________________________________________
Name: svn:executable
+ *
Added: ESI-Examples/sys/scripts/eg-wal-archive.sh
===================================================================
--- ESI-Examples/sys/scripts/eg-wal-archive.sh (rev 0)
+++ ESI-Examples/sys/scripts/eg-wal-archive.sh 2010-03-17 17:57:36 UTC (rev 835)
@@ -0,0 +1,92 @@
+#!/bin/bash
+#!/bin/bash
+# Evergreen WAL archiving script
+# Copyright (C) 2008-2010 Equinox Software Inc.
+# Mike Rylander <mrylander at gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+#
+# You will need to edit the variables below to configure this script for
+# use in your production environment.
+#
+# If WAL file shipping is enabled then the postgres user should be able to
+# log into the remote host as the ARCHIVE_USER over ssh and scp with a
+# passphraseless ssh key.
+#
+# In your postgresql.conf, turn on archive_mode (if applicable) and adjust
+# the archive_command thusly:
+#
+# archive_command = '/location/of/this/script/eg-wal-archive.sh %p %f'
+#
+
+
+#--------------------- CONFIGURATION BEGIN ------------------------
+# Remove the following line once you have adjusted the configuration
+# below to match your environment.
+echo "Configuration not complete!" && exit 1;
+
+
+# File which, if it exists, pauses WAL archiving
+PAUSE_FILE=/tmp/wal-pause
+
+# Local WAL archiving directory
+ARCHIVE_DIR="/var/backup/$HOSTNAME/evergreen/database/wal/"
+
+
+# Remote host (IP or resolvable name) to which WAL files should be shipped.
+# Leave empty to disable WAL file shipping.
+ARCHIVE_HOST=
+
+
+# User on the remote WAL-receiving host.
+ARCHIVE_USER=
+
+# Snapshot archiving directory on the remote host, if WAL file shipping is
+# enabled.
+ARCHIVE_DST="/var/backup/$HOSTNAME/evergreen/database/wal/"
+
+#--------------------- CONFIGURATION END ------------------------
+
+
+
+
+while [ -e $PAUSE_FILE ]; do sleep 1; done
+
+P=$1
+F=$2
+
+if [ -e $ARCHIVE_DIR/$F.bz2 ]; then
+ echo "Cannot archive: $ARCHIVE_DIR/$F.bz2 already exists"
+ logger -p local3.info "Cannot archive: $ARCHIVE_DIR/$F.bz2 already exists"
+ exit 0;
+fi
+
+cp $P $ARCHIVE_DIR/$F
+CP_RES=$?
+
+if [ "$CP_RES" != "0" ]; then
+ echo "Cannot archive: unable to copy WAL file $P to $ARCHIVE_DIR/$F, cp exit code = $CP_RES"
+ logger -p local3.info "Cannot archive: unable to copy WAL file $P to $ARCHIVE_DIR/$F, cp exit code = $CP_RES"
+ exit 1;
+fi
+
+/bin/bzip2 $ARCHIVE_DIR/$F
+if [ "_$ARCHIVE_HOST" != "_" ]; then
+ scp -q $ARCHIVE_DIR/$F.bz2 $ARCHIVE_USER@$ARCHIVE_HOST:$ARCHIVE_DST
+fi
+
+exit 0
+
Property changes on: ESI-Examples/sys/scripts/eg-wal-archive.sh
___________________________________________________________________
Name: svn:executable
+ *
More information about the open-ils-commits
mailing list