Mysql backup script with date and time stamp to filename

Posted on September 4, 2011 by admin

Here is how to dump all mysql database tables while gzip’ing at the same time. Then backing up to amazong s3 with the addition of a time stamp to the filename.

#! /bin/bash
BACKUP_PATH=’/var/backups/mysql’
cd $BACKUP_PATH
FILENAME=$(hostname).`date +%Y%m%d`_mysql

PATH=/usr/local/jdk/bin:/usr/local/jdk/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/X11R6/bin:/home/myjobapp/bin:/usr/local/bin:/usr/X11R6/bin

rm -rf *.gz
/usr/bin/mysqldump –opt –user=dbuser –password=dbpass –all-databases | gzip -9 > $FILENAME.sql.gz
The result creates this filename:
db1.rezik.net.20110904_mysql.sql.gz

Furthermore, when I copy the mysql dump file to my cloud backup server I do this:

#! /bin/bash
s3cmd put /var/backups/mysql/db1.rezik.net.`date +%Y%m%d`_mysql.sql.gz s3://rezik.net/db1/db1.rezik.net.`date +%Y%m%d`_mysql.sql.gz | mail -s “db1-rezik.net-bkup” backups@rezik.net

Posted in Liber Linux | Leave a comment

Resize Logical Volume

Posted on September 3, 2011 by admin

Step 1:
unmount the disk / If you are using xen shutdown the vm

Step 2:
Example of adding 100GB
lvextend -L100G /dev/xen-images/reziktech-disk
or
lvresize -L+100G /dev/xen-images/reziktech-disk

Removing 100GB
lvresize -L-100G /dev/xen-images/reziktech-disk

check it:
cat /dev/xen-images/reziktech-disk

— Logical volume —
LV Name /dev/xen-images/reziktech-disk
VG Name xen-images
LV UUID tlbABd-BZ3t-ITGD-YO8L-qUFL-rRGf-LIooUG
LV Write Access read/write
LV Status available
# open 1
LV Size 600.00 GB <—good it was 500GB
Current LE 1280
Segments 1

Mount the disk or start the vm

Step 3:
resize2fs /dev/sda2 (enter this command within the vm that you just resized)

Verify it:
df -h

Posted in Liber Linux | Tagged , , , , | Leave a comment

Postfix queue is backed up

Posted on September 3, 2011 by admin

Check how many emails are in the queue:
# mailq |wc -l

To remove all mail from the queue:
# postsuper -d ALL

To remove all deferred email from the queue:
# postsuper -d ALL deferred

Posted in Liber Linux | Tagged , , , | Leave a comment

very basic apf firewall setup

Posted on July 1, 2011 by admin
Connect to your servers terminal.  cd /opt or where ever you choose.  su root
wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz

tar zxvf apf-current.tar.gz

cd apf-your-version

./install.sh

Installation Details:
Install path:         /etc/apf/
Config path:          /etc/apf/conf.apf
Executable path:      /usr/local/sbin/apf
Other Details:
Listening TCP ports: 22,25,2000,6010  Listening UDP ports: 2727,4520,4569,5060
Note: These ports are not auto-configured; they are simply presented for information purposes. You must manually configure all port options.

First thing I do is enter the host IP’s for the system admins, allowing * from admins IP.  I whitelist our IP’s rather than opening up ssh port 22 and mysql port 3306 to the world.  This way the server wont even answer to a potential cracker.  Obviously this setup wont apply to everyone so adjust the config to fit your needs.

vi /etc/apf/allow_hosts.rules

Using a new line for each ip address, add the addresses you wish to whitelist to allow_hosts.rules.  To only allow specific ports for a specific host use the format below.
tcp:in:d=3306:s=10.1.2.45
tcp:in:d=22:s=10.3.3.29

Now, setup conf.apf

vi /etc/apf/conf.apf

change DEVEL_MODE=”1″ to DEVEL_MODE=”0″

The ports below are added by default except 5060 i added for VoIP.
# 8: Ports for Maximum Throughput – Minimum Delay
TOS_8=”21,20,80,5060″
# 16: Ports for No Delay – Moderate Throughput – High Reliability
TOS_16=”25,110,143″

Starting APF

/etc/init.d/apf (start, stop, restart)

Documentation is available in the file README.apf in the directory where apf was extracted.

Posted in Liber Linux | Tagged , , , , , , , , | Leave a comment

mysql users

Posted on June 29, 2011 by admin

Create initial mysql user post install:
mysqladmin -u root password newpass
Update a password using mysqladmin:
mysqladmin -u root -p’oldpass’ password newpass

Create mysql user:
mysql>CREATE USER ‘rion’@'localhost’ IDENTIFIED BY ‘mypass’;

Set password for mysql user:
mysql>SET PASSWORD FOR ‘rion’@'localhost’ = PASSWORD(‘newpass’);

Grant privileges to mysql user:
Normal user -
mysql>GRANT INSERT,SELECT,UPDATE,PROCESS ON *.* TO ‘rion’@'localhost’;
Admin user -
mysql>GRANT INSERT,SELECT,UPDATE,PROCESS,DELETE,CREATE,SUPER ON *.* TO ‘rion’@'locahost’;

Delete user:
mysql>DROP USER ‘rion’@'localhost’;

After editing any mysql user finalize by entering:
mysql>FLUSH PRIVILEGES;

Posted in Liber Linux | Tagged , , , , , | Leave a comment

remove SSL pass phrase

Posted on June 28, 2011 by admin

Apache halts on start up until the pass phrase is authenticated.  This can be a real problem for some people that may have rebooted their Virtual Server and it all came back up except Apache2 which is waiting for user input on the console.  At that point someone can either access the server’s host controller, then your servers virtual console, then enter the pass phrase.  Or most commonly just ssh to server and kill apache and start manually from user shell.  This is real beginner talk…

Problem:
Apache has halted and is asking for pass phrase each time it starts.

Solution:
have your pass phrase ready and locate your servers ssl certificate key file.
SSLCertificateKeyFile /etc/ssl/private/rezik.key

openssl rsa -in /etc/ssl/private/rezik.key -out /etc/ssl/private/rezik.key

Posted in Liber Linux | Tagged , , , , | Leave a comment

SSL on debiAn

Posted on June 28, 2011 by admin

The commands below will generate a 2048 bit key then the csr which you will need to upload to your ssl host (godaddy, geotrust, etc..)

openssl genrsa -des3 -out <name of your certificate>.key 2048

openssl req -new -key <name of your certificate>.key -out <name of your certificate>.csr

Posted in Liber Linux | Tagged , , | Leave a comment

lan + cloud backup hub

Posted on June 28, 2011 by admin

This article and proposed solution acts as an extension from the s3 backup post and also the rsync post.  Please use those articles in conjunction with this one.

I created a virtual machine as my backup ‘hub’ giving me a local copy of everything and then only the ‘hub’ uses s3cmd.  On each of my servers i simply added the user backupadmin then use an rsync pull method to get the data from each server, where it is then sync’d to my s3 bucket.  Here are the steps for that:

on backup hub: ssh-keygen (this will create an ssh key and store it in ~/.ssh/)
on each other server: adduser backupadmin
on backup hub: ssh-copy-id -i ~/.ssh/id_rsa.pub backupadmin@192.168.1.112

Perform the second and third steps for each system you want to backup.  ssh-copy-id will automatically copy the key to the correct location on the remote server.  Now you do not need to enter a password for rsync.  Only rsyncd allows a password file.  Now we will setup the command to pull the data to the backup hub.  Create the destination directories for the data.  I use the servername then the exact folders being backup up for uniformity.  For example: mkdir  ~/server1 then mkdir ~/server1/public_html to backup the public_html directory on server1.

rsync -avz -i –bwlimit=963 –log-file=/home/backupadmin/logs/server1-public_html.log backupadmin@192.168.1.112:/home/rion/public_html/ ~/server1/public_html

I add that command to a cron that runs every night at midnight.  Then two hours later the cron to backup the folder to s3.

s3cmd -p sync /home/backupadmin/server1/public_html/ s3://server1/public_html/

Posted in Liber Linux | Tagged , | Leave a comment

rsync

Posted on June 28, 2011 by admin

This article shows how to sync two directories either locally or on a remote host using rsync; a fast, versatile, remote (and local) file-copying tool. This works any direction you need it to, meaning rsync can be initiated from either the source or destination system or likewise the source or destination directory.  This is called pushing and pulling depending on which direction you are performing the copy. Just keep in mind the most important rule with rsync is the order of operations which is always rsync [options] source.. [destination].

Local:  rsync [OPTION...] SRC… [DEST]
Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC… [DEST]
Push: rsync [OPTION...] SRC… [USER@]HOST:DEST
(see: man rsync)

If you do not have rsync installed on your system open a terminal and enter:

sudo apt-get install rsync

Examples:

1. Copy all contents of a directory on a remote system over ssh to a local folder verbose, preserving permissions, user/group, and using compression. Initiating the rsync from my home computer.

rsync -avz rion@rezik.net:/home/reziknet/public_html/ /home/rion/reziknet/public_html/

Make note of the slashes at the end, this does matter, the trailing slash means to copy the entire contents of public_html without copying the directory public_html since i already have it on my local machine.  The same could be accomplished if i remove the slashes and set my destination to /home/rion/reziknet/.  Very important to watch your slashes!

2. The same sync as example 1 with the addition of a bandwidth limit and log file output using push rather than pull.  Initiating the rsync from the server.

rsync -avz -i –bwlimit=600 –log-file=rsync.log  /home/reziknet/public_html/ rion@homeip.com:/home/rion/reziknet/public_html/

When i am performing an rsync during business hours i limit the bandwidth so other people using the connection aren’t effected.

3. Copying files from one directory to another on the local machine.

rsync -av /home/reziknet/ /var/backups/2010/3/reziknet/

4. Copying files using *

rsync -avz rion@rezik.net:/home/reziknet/public_html/*.php /home/rion/phpfiles/

rsync -av /var/log/apache/ap* /var/backups/2010/3/logs/

Definition of options used in examples.
-a, –archive               archive mode; equals -rlptgoD (see man page)
-v, –verbose               increase verbosity
-z, –compress              compress file data during the transfer
-i, –itemize-changes       output a change-summary for all updates
–bwlimit=KBPS          limit I/O bandwidth; KBytes per second
–log-file=FILE         log what we’re doing to the specified FILE

The -a option basically takes care of everything you’ll need, option -v gives a detailed output of what exactly its doing, you may not need to use -z for compression however i feel that it is good practice when rsync’ing over a network, -i is required when adding itemized changes to the action.  There are an arsenal of options in rsync, basically if you can think of it rsync can do it.

The complete manual for rsync is found by typing ‘man rsync’ into your terminal.
rsync’ing: verb -The act of syncing directories on a linux system.
Enjoy!

Posted in Liber Linux | Tagged , , , , , | Leave a comment

amazon s3 backup with email reports

Posted on June 28, 2011 by admin

There are a variety of tools out there that include solutions for backing up to amazon s3 however none of them are as easy and straight forward as the debian package s3cmd.  I tried everything before trying s3cmd and it was a mistake so i recommend not wasting your time and just start off right.  s3cmd is very easy to install, configure, and use.  For automatic backups just drop your s3cmd command into a cron and your set.  Many people try going with the s3snc.rb ruby script but it requires way more effort and packages to install when really all it does is backup using s3cmd.

The first thing i did was create a user called backupadmin and gave the user appropriate permissions to the locations i wanted to backup.  I recommend doing that however it is not required.  s3cmd can run under any user and you can configure multiple s3cmd users.  When you configure s3cmd it will create a file .s3cmd in the users home directory in which the s3 settings will be stored.  To install s3cmd use the following command as super user

apt-get install s3cmd

Now that it is installed you need to configure it.

s3cmd –configure

You will be prompted for the access key and secret key you can find/generate from your s3 account portal.  You will also be prompted to use ssl/https for your data transfer.  It is slower so if you don’t have security sensitive data then definitely enter no for https.  If you are backing up secure data answer yes to https.  In my case I had both secure and public data so i setup another user for the https backups ‘sbackupadmin’ for secure backup admin then ran the configuration again under that user.

After running s3cmd –configure a file is created in your home directory called .s3cfg.  If you run configure as root then you will find the config file in /root/.s3cfg.  Once you run configure you can use the .s3cfg file generated as a template to copy to other users/servers.  .s3cfg is easily edited to make adjustments such as tuning your ssl and gpg settings.

Here are a few examples of how I use s3cmd.

To create a remote folder ‘bucket’ and put a single file in the new bucket:

s3cmd mb s3://server1
s3cmd mb s3://server1/mysql
s3cmd -p put /home/backupadmin/server1/mysqldump.tar.gz s3://server1/mysql/
s3cmd ls s3://server1/mysql (this lists the remote bucket to see the file was copied)

To sync a folders contents to a remote bucket (like rsync):

s3cmd -p sync /home/backupadmin/server2/public_html/ s3://server2/public_html

When using sync after the files are copied initially only new files and files modified since last s3 backup will be copied.  This is an awesome solution for almost all backup solutions.  Even if i am doing one single file i use sync.  The -p option there preserves the user/group and permissions.  You will probably always want to use the -p option.

A simple way to backup all of your servers is to simply drop your s3cmd command into a cron on each server and let it rip!

Warning:
you probably do not want to run both https and encryption or it will run very slow
–check your .s3cfg file and edit accordingly

Advanced backup section:
s3cmd does not handle large directories well!  If you have more than 200 folders in your command you will want to break it up and do one folder at a time using a shell script.  In my case I am working with a public_html folder with 1800 folders and growing.  Here is what I did to not only solve the resources and speed issue but also allows control of backup strength, 1X, 2X, 3X, etc..

1. ls /home/example/public_html/ > folder-list.txt

2. Create open office calc spreadsheet and paste folder-list.txt in column B

3. Sequentially  (s3cmd -p sync /home/example/public_html/)paste beginning of command in column A, then populate cells down even with last of column B

4. In column C enter a unique character such as $ and drag down to last row in set (used for search & replace later to add a space in command)

5. paste destination in column D and drag down to bottom row of set (s3://example/public_html/)

5. copy all column B to column E for destination folder

6. highlight entire data set and paste in gedit

7. search and replace ‘tab’ gaps to nothing – remove tab spaces from fields then replace $ with ‘ ‘ a space.

Result:
s3cmd -p sync /home/example/public_html/client1 s3://example/public_html/client1
s3cmd -p sync /home/example/public_html/client2 s3://example/public_html/clinet2
s3cmd -p sync /home/example/public_html/client3 s3://example/public_html/client3
s3cmd -p sync /home/example/public_html/file.php s3://example/public_html/file.php

Name the file something like s3-pub_html.sh and copy to /usr/local/bin on server.

chmod +x /usr/local/bin/s3-pub_html.sh

At this point you are good to go. Depending on how big your backup is you may or may not need the next step.  The next step will break up the script into chunks that can be run in parallel for 2X, 3X+ backups.  The example below I am doing a 2X backup.

8. For Split if you would like to run scripts since i have 1827 lines. 1827 divided by 8 = 228.375  – split by 225 then put leftovers in last file

split -225 s3-pub_html.sh

List the directory and you will see the split files xaa, xab, xac, etc each containing 225 lines except the last one with the remainder.  rename those files to xaa.sh, xab.sh, etc.  create 2 new master scripts and add the split scripts to them

s3-pub_html-1x.sh
sh /usr/local/bin/xaa.sh | mail -s “s3-pub_html-bkup1″ s3report@rezik.net
sh /usr/local/bin/xab.sh | mail -s “s3-pub_html-bkup2″ s3report@rezik.net
sh /usr/local/bin/xac.sh | mail -s “s3-pub_html-bkup3″ s3report@rezik.net
sh /usr/local/bin/xad.sh | mail -s “s3-pub_html-bkup4″ s3report@rezik.net

s3-pub_html-2x.sh
sh /usr/local/bin/xaa.sh | mail -s “s3-pub_html-bkup5″ s3report@rezik.net
sh /usr/local/bin/xab.sh | mail -s “s3-pub_html-bkup6″ s3report@rezik.net
sh /usr/local/bin/xac.sh | mail -s “s3-pub_html-bkup7″ s3report@rezik.net
sh /usr/local/bin/xad.sh | mail -s “s3-pub_html-bkup8″ s3report@rezik.net

crontab example for 2X
15 22 * * * sh /usr/local/bin/s3-pub_html-1x.sh
0 0 * * * sh /usr/local/bin/s3-pub_html-2x.sh

The first backup begins at 11:15pm and then the second session starts at midnight.  This allows our backups to finish by 4:00am where it would take until 8:00am at 1X.

Below is the details for s3cmd.  You can get it by typing s3cmd –help in your command line.  enjoy!

Usage: s3cmd [options] COMMAND [parameters]

S3cmd is a tool for managing objects in Amazon S3 storage. It allows for
making and removing “buckets” and uploading, downloading and removing
“objects” from these buckets.

Options:
-h, –help            show this help message and exit
–configure           Invoke interactive (re)configuration tool.
-c FILE, –config=FILE
Config file name. Defaults to /root/.s3cfg
–dump-config         Dump current configuration after parsing config files
and command line options and exit.
-e, –encrypt         Encrypt files before uploading to S3.
–no-encrypt          Don’t encrypt files.
-f, –force           Force overwrite and other dangerous operations.
-P, –acl-public      Store objects with ACL allowing read for anyone.
–acl-private         Store objects with default ACL allowing access for you
only.
–delete-removed      Delete remote objects with no corresponding local file
[sync]
–no-delete-removed   Don’t delete remote objects.
-p, –preserve        Preserve filesystem attributes (mode, ownership,
timestamps). Default for [sync] command.
–no-preserve         Don’t store FS attributes
–exclude=GLOB        Filenames and paths matching GLOB will be excluded
from sync
–exclude-from=FILE   Read –exclude GLOBs from FILE
–rexclude=REGEXP     Filenames and paths matching REGEXP (regular
expression) will be excluded from sync
–rexclude-from=FILE  Read –rexclude REGEXPs from FILE
–debug-syncmatch, –debug-exclude
Output detailed information about remote vs. local
filelist matching and –exclude processing and then
exit
–bucket-location=BUCKET_LOCATION
Datacentre to create bucket in. Either EU or US
(default)
-m MIME/TYPE, –mime-type=MIME/TYPE
Default MIME-type to be set for objects stored.
-M, –guess-mime-type
Guess MIME-type of files by their extension. Falls
back to default MIME-Type as specified by –mime-type
option
-H, –human-readable-sizes
Print sizes in human readable form.
-v, –verbose         Enable verbose output.
-d, –debug           Enable debug output.
–version             Show s3cmd version (0.9.8.3) and exit.

Commands:
Make bucket
s3cmd mb s3://BUCKET
Remove bucket
s3cmd rb s3://BUCKET
List objects or buckets
s3cmd ls [s3://BUCKET[/PREFIX]]
List all object in all buckets
s3cmd la
Put file into bucket
s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
Get file from bucket
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Delete file from bucket
s3cmd del s3://BUCKET/OBJECT
Synchronize a directory tree to S3
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
Disk usage by buckets
s3cmd du [s3://BUCKET[/PREFIX]]
Get various information about Buckets or Objects
s3cmd info s3://BUCKET[/OBJECT]

See program homepage for more information at

http://s3tools.logix.cz

Posted in Liber Linux | Tagged , , , , | Leave a comment