Wednesday, March 28, 2012

使用 rsnapshot 備份及比較設定檔

1. 安裝 rsnapshot
在 Ubuntu 下直接用 apt-get install rsnapshot.
在 FreeBSD 下請到 /usr/ports/sysutils/rsnapshot 下 make install clean. (在我的 FreeBSD 實體及虛擬環境中並沒有實際安裝測試, 以下僅用 Ubuntu 作為範例.)

2. 調整設定檔 /etc/rsnapshot.conf
# diff /etc/rsnapshot.conf /etc/rsnapshot.conf.20120322
27c27
< snapshot_root /.rsnapshot/
---
> snapshot_root /var/cache/rsnapshot/
97c97
< retain                hourly  8
---
> retain                hourly  6
100c100
< retain                monthly 6
---
> #retain       monthly 3
124c124
< logfile       /var/log/rsnapshot.log
---
> #logfile      /var/log/rsnapshot.log
230c230
< #backup       /home/          localhost/
---
> backup        /home/          localhost/
232,235c232
< backup        /usr/local/etc/ localhost/
< backup        /var/log/       localhost/
< backup        /var/mail/      localhost/
< backup        /root           localhost/
---
> backup        /usr/local/     localhost/
註 : 因為 /home 已將大部分資料放上 NFS, 已經有 ZFS snapshot. 就不浪費時間多做 snapshot.

3. 設定定期執行
0 */3 * * * /usr/bin/rsnapshot hourly
0 23 * * * /usr/bin/rsnapshot daily
15 23 * * 1 /usr/bin/rsnapshot weekly
30 23 1 * * /usr/bin/rsnapshot monthly
0 0 * * 1 /PATH/TO/REPORT/PROG/report_CHK_ETC_diff.sh

4. CHK_ETC_diff.sh
#!/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        CHKPATH="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SSKEYWORD="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SSBASE="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 check_path snapshot_keyword snapshot_base_path"
        echo "Example: $0 /etc Weekly /SNAPSHOT/System"
        exit 1
else
        echo "Collect difference between $CHKPATH and $SSBASE/${SSKEYWORD}*$CHKPATH."
fi

date +"%Y/%m/%d %H:%M:%S"
LAST_DIR="/"
CHKPATH=`echo $CHKPATH | sed -e 's/\///'`

for CONF in `find $LAST_DIR$CHKPATH -type f -name '*'`
do
        if [ `ls $SSBASE | grep ${SSKEYWORD} |wc -l` -lt 1 ]; then
                echo "$SSBASE/${SSKEYWORD}* does not existed!"
                exit 1
        fi
        ls -d $SSBASE/${SSKEYWORD}* | while read DIR
        do
                if [ $LAST_DIR = "/" ]; then
                        HOST=""
                else
                        HOST="/localhost"
                fi
                if [ -f ${LAST_DIR}${HOST}${CONF} ]&&[ -f ${DIR}/localhost${CONF} ]; then
                        if [ $LAST_DIR = "/" ]; then
                                LAST_DIR=""
                        fi
                        DIFFC=`diff ${LAST_DIR}${HOST}${CONF} ${DIR}/localhost${CONF} | wc -l`
                        if [ $DIFFC -gt 0 ]; then
                                echo "[[[ Difference between ${LAST_DIR}${HOST}${CONF} and ${DIR}/localhost${CONF} ]]]"
                                diff ${LAST_DIR}${HOST}${CONF} ${DIR}/localhost${CONF}
                                echo ""
                        fi
                        DIFFC=0
                else
                        if [ ! -f ${LAST_DIR}${HOST}${CONF} ]; then
                                echo "File ${LAST_DIR}${HOST}${CONF} does not existed."
                        fi
                fi

                LAST_DIR=$DIR
        done
done
echo ""
date +"%Y/%m/%d %H:%M:%S"

5. report_CHK_ETC_diff.sh
#!/bin/bash

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:${PATH}

CHKPROG="/PATH/TO/PROG/CHK_ETC_diff.sh /etc daily /.rsnapshot"
REPORTPATH="/PATH/TO/STORE/REPORT/report_CHK_ETC_diff"
REPORTKEYWORD="etcdiff"
REPORTTITLE="Check Difference of ETC Report"
REMAILADDR="ACCOUNT@DOMAIN.NAME"

mkdir -p $REPORTPATH/`date +"%Y"`
mv $REPORTPATH/${REPORTKEYWORD}_*.log $REPORTPATH/`date +"%Y"`
$CHKPROG > $REPORTPATH/${REPORTKEYWORD}_`hostname`_`date +"%Y%m%d"`.log
cat $REPORTPATH/${REPORTKEYWORD}_`hostname`_`date +"%Y%m%d"`.log | mail -s "$REPORTTITLE" $REMAILADDR

參考資料:
淺談linux下的快照備份軟體—rsnapshot

Wednesday, March 21, 2012

應用 UFS snapshot 進行比對 /etc 及 /usr/local/etc 下的檔案內容

1. 這隻程式運作的前提是已經完成 UFS snapshot 並且已經正常掛載. 沒有掛載 snapshot 只會看到出現很多找不到檔案的訊息.

2. 程式碼
#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        CHKPATH="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SSKEYWORD="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SSBASE="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 check_path snapshot_keyword snapshot_base_path"
        echo "Example: $0 /etc Weekly /SNAPSHOT/System"
        exit 1
else
        echo "Collect difference between $CHKPATH and $SSBASE/${SSKEYWORD}*$CHKPATH."
fi

date +"%Y/%m/%d %H:%M:%S"
LAST_DIR="/"
CHKPATH=`echo $CHKPATH | sed -e 's/\///'`

for CONF in `find $LAST_DIR$CHKPATH -type f -name '*'`
do
        if [ `ls $SSBASE | grep ${SSKEYWORD} |wc -l` -lt 1 ]; then
                echo "$SSBASE/${SSKEYWORD}* does not existed!"
                exit 1
        fi
        ls -d $SSBASE/${SSKEYWORD}* | while read DIR
        do
                if [ -f ${LAST_DIR}${CONF} ]&&[ -f ${DIR}${CONF} ]; then
                        if [ $LAST_DIR = "/" ]; then
                                LAST_DIR=""
                        fi
                        DIFFC=`diff ${LAST_DIR}${CONF} ${DIR}${CONF} | wc -l`
                        if [ $DIFFC -gt 0 ]; then
                                echo "[[[ Difference between ${LAST_DIR}${CONF} and ${DIR}${CONF} ]]]"
                                diff ${LAST_DIR}${CONF} ${DIR}${CONF}
                                echo ""
                        fi
                        DIFFC=0
                else
                        if [ ! -f ${LAST_DIR}${CONF} ]; then
                                echo "File ${LAST_DIR}${CONF} does not existed."
                        fi
                fi

                LAST_DIR=$DIR
        done
done
echo ""
date +"%Y/%m/%d %H:%M:%S"

3. 使用範例
/PATH/TO/CHK_ETC_diff.sh /PATH/TO/SRC SNAPSHOT_KEYWORD /PATH/TO/SNAPSHOT_ROOT

Sunday, March 18, 2012

在 FreeBSD 8 對 UFS filesystem 做 snapshot

1. 在 crontab 定期做 snapshot
### UFS snapshot
0    1      *       *       *       root    /usr/local/sbin/snapshot make -g7 /SRCPATH:Daily
0      2      *       *       1       root    /usr/local/sbin/snapshot make -g4 /SRCPATH:Weekly

2. 掛載 UFS snapshot 的 shell script
#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        SSSRCPATH="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SnapshotName="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        MountPath="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 Snapshot_Src_Path SnapshotName MountPath"
        echo "Example: $0 /:Weekly.0 /mnt"
        exit 1
else
        echo "Try to mount $SSSRCPATH:$SnapshotName to $MountPath now."
fi

MC=`df -h |grep $MountPath |wc -l`

if [ -d $SSSRCPATH ]; then
        if [ -f $SSSRCPATH/.snap/$SnapshotName ]; then
                if [ -d $MountPath ]; then
                        if [ $MC -gt 0 ]; then
                                echo "MountPath $MountPath had been mounted!"
                        else
                                snapshot mount $SSSRCPATH:$SnapshotName $MountPath
                                df -h |grep $MountPath
                                echo "MountPath $MountPath was mounted."
                        fi
                else
                        echo "MountPath $MountPath does not existed!"
                fi
        else
                echo "Snapshot $SSSRCPATH/.snap/$SnapshotName does not existed!"
        fi
else
        echo "Snapshot source path $SSSRCPATH does not existed!"
fi

3. 掛載範例
# mkdir -p /SNAPSHOT/Mount/Path/Weekly0

/PATH/TO/mount_UFS_snapshot.sh /SRCPATH Weekly.0 /SNAPSHOT/Mount/Path/Weekly0

4. 卸載 UFS snapshot 的 shell script
#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        MountPath="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 MountPath"
        echo "Example: $0 /mnt"
        exit 1
else
        echo "Try to umount $MountPath now."
fi

MC=`df -h |grep $MountPath |wc -l`

if [ -d $MountPath ]; then
        if [ $MC -gt 0 ]; then
                snapshot umount $MountPath
                df -h |grep $MountPath
                echo "MountPath $MountPath had been umounted!"
        else
                echo "MountPath $MountPath had not been mounted!"
        fi
else
        echo "MountPath $MountPath does not existed!"
fi

5. 卸載範例
/PATH/TO/umount_UFS_snapshot.sh /SNAPSHOT/Mount/Path/Weekly0

6. UFS 做 snapshot 速度還蠻慢的. (相較於 ZFS 的快速完成 snapshot)

參考資料
FreeBSD UFS/ZFS Snapshot Management Environment

Saturday, March 03, 2012

在 FreeBSD 及 Ubuntu 啟動 MLS SSD 支援 TRIM

目前手邊的機器每一台都至少裝了一顆 MLC SSD. 不論是當系統碟或當暫存區使用, 都有需要啟動 TRIM. 剛好 FreeBSD 8.2 及 Ubuntu 11.10 的 kernel 都有支援 TRIM. 所以就參考文件設定.

[[[ FreeBSD 8.2 ]]]
1. Reboot and login in Single User Mode
2. tunefs -p /dev/adXXsXX
3. tunefs -t enable /dev/adXXsXX
4. Reboot the machine
5. Check the setting
HHHHH(XXXXXX)/root# tunefs -p /dev/adXsXX
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         enabled
tunefs: maximum blocks per file in a cylinder group: (-e)  2048
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

[[[ Ubuntu 11.10 ]]]
1. Edit /etc/fstab and add "discard,noatime,nodiratime,".
root@HOSTNAME:~# diff /etc/fstab.bak /etc/fstab
10c10
< UUID=dXXXdXXX-5899-XXXX-XXXd-XXXXcXXX8XXX /               ext4    errors=remount-ro 0       1
---
> UUID=dXXXdXXX-5899-XXXX-XXXd-XXXXcXXX8XXX /               ext4    discard,noatime,nodiratime,errors=remount-ro 0       1

2. Edit /etc/rc.local and add following statements before 'exit 0'.
### For tweak SSD
echo noop > /sys/block/sda/queue/scheduler
echo 1 > /sys/block/sda/queue/iosched/fifo_batch

3. Reboot the machine

4. 參考這篇做測試. HOWTO: Check If TRIM On Ext4 Is Enabled And Working On Ubuntu And Other Distributions


參考資料 Reference:
TRIM support on UFS2 FreeBSD 9
Racklin's 阿土伯程式大觀園: 為 Thinkpad T61 Debian Linux 換上 SSD
HOWTO: Configure Ext4 to Enable TRIM Support for SSDs on Ubuntu and Other Distributions
HOWTO: Check If TRIM On Ext4 Is Enabled And Working On Ubuntu And Other Distributions
How to Tweak Your SSD in Ubuntu for Better Performance

在 FreeBSD, PC-BSD, Ubuntu, Mint, CentOS, Fedora, OpenSUSE, Debian 上使用 tmpfs 掛載 /tmp

首先說明一下, 以下是給懶人專用, RAM Disk 大小都是預設最大到 50% 記憶體. 如果有需要調整空間使用上限或在大型伺服器上實做, 請務必研究一下附件還有 tmpfs. 另外 NetBSD 5, OpenSolaris 及 Indiana 就不用傷腦筋了, 系統預設已經使用 tmpfs(在 NetBSD 使用 mfs) 掛載 /tmp.

由於現代的機器的記憶體基本上都是 GB 等級. 讓記憶體閒著也是一種浪費. 而更重要的原因是不想讓讀寫 /tmp 的動作大量存取 MLC SSD(雖然已經啟動了 TRIM). 再加上 guest VM 常常讀寫, 多跑幾台對 VM host 還是有效能上的影響. 就這樣看著大家的文件跟著做. 也順手將手邊的測試用 guest VM 加上這個簡單的設定. 以下就是實際上在各平台上的設定方式:

[[[ FreeBSD 8, PC-BSD 8 & 9 beta ]]]
1. Edit /etc/fstab
tmpfs  /tmp  tmpfs  rw 0 0

2. Reboot the machine

[[[ Ubuntu 11.X 12.04 LTS, Mint 11 ]]]
1. Edit /etc/init.d/ramtmp.sh
#!/bin/sh
# RamDisk tmp
PATH=/sbin:/bin:/usr/bin:/usr/sbin

mkdir /dev/shm/tmp
chmod 1777 /dev/shm/tmp
mount --bind /dev/shm/tmp /tmp

2. chmod 755 /etc/init.d/ramtmp.sh
3. cd /etc/rcS.d; ln -s /etc/init.d/ramtmp.sh SXXramtmp
XX : 必須在所有需要 /tmp 目錄的服務之前. 請注意英文字母排序.
簡單的作法就是在 /etc/rcS.d 下 grep tmp ./* 看有哪些檔案需要 /tmp 目錄.
4. Reboot the machine

[[[ CentOS 6, Fedora 15, ScientificLinux 5.6 ]]]
1. Edit /etc/rc.d/init.d/ramtmp
#!/bin/bash
# chkconfig: 35 80 70
# description: RAMDisk4Tmp
#
# History:
### 2012/02/25 RamDisk tmp

PATH=/sbin:/bin:/usr/bin:/usr/sbin

mkdir /dev/shm/tmp
chmod 1777 /dev/shm/tmp
mount --bind /dev/shm/tmp /tmp

2. chmod 755 /etc/rc.d/init.d/ramtmp
3. chkconfig --add ramtmp
4. Reboot the machine

[[[ OpenSUSE 11.4 ]]]
1. Edit /etc/fstab
tmpfs  /tmp  tmpfs  defaults 0 0

2. Reboot the machine

[[[ Smoothwall express 3.0 sp2 ]]]
1. Edit /etc/fstab
none  /tmp  tmpfs  defaults 0 0

2. Reboot the machine

[[[ ArchLinux 3.0 ]]]
1. Edit /etc/fstab
tmpfs  /tmp  tmpfs  nodev,nosuid 0 0

2. Reboot the machine

[[[ Debian 6 ]]]
1. Edit /etc/fstab
tmpfs /tmp tmpfs defaults,nodev,nosuid 0 0

2. Reboot the machine

[[[ Windows 7 Ultimate, Windows server 2003, Windows XP ]]]
請安裝 Dataram_RAMDisk_V3.5.130R22, 詳細步驟請看參考資料.

參考資料 Reference:
FreeBSD : 【FreeBSD】用 tmpmfs 或 tmpfs 建立 Ramdisk
Ubuntu : Firefox & Chrome Cache on RAM Drive -Fedora / Ubuntu
                 在 Ubuntu 將 Firefox 網頁快取檔,設到 Ramdisk 的方法
CentOS : [筆記]Linux Tmpfs 系統加速實作
OpenSUSE : SDB:SSD performance
Debian : HOWTO: Using tmpfs for /tmp
Windows : [桌面系統]Windows 7 (32bit/64bit) 使用RamDisk 外加ReadyBoost 增加效能