Scripting 101: Purging Files

Ei-lo! 'Zup? I'm beginning to enjoy this I guess: scripting. Unexpected, I got a ping from a former colleague, asking from a noobs [read: me] regarding scripting. Without blinking, I accepted the challenge. I was asked to check on the script she made for purging files. After checking on the requirements, given, and asking for advise from a scripting guru [Morsgang Freeman], below are the revised forms. Btw, the first one I made really sucks [I hope these won't :(]. As of writing, these are not tested, not 'til tomorrow when she receives the email/script.

1. First is to create a file, say SUBDIR.txt with the ff. contents:

MERGE/REJECTS/ARCHIVE
MERGE/REJECTS/ARCHIVERAW
MERGE/FILTERED/ARCHIVE
MERGE/FILTERED/ARCHIVERAW
INSERT/REJECTS/INSERTED
INSERT/FILTERED/INSERTED
FTP/FTP_ARCHIVE

2. I made two variations of the script:
a. Version 1

#!/bin/bash

export LOGDIR=/ramsys_data/bmcps/ramsys/
export DIRLIST="ALTL AMA5 CAMA FREE NRTL SEGR UTSS"

for XXX in DIRLIST
do
cd $LOGDIR
if [ "$PWD" = "$LOGDIR" ]
then
for PURDIR in `cat SUBDIR.txt`
do
# A log file can be added here [good practice] to list the files to be deleted...
find $XXX/$PURDIR -type f -name "*" -mtime +5 -exec rm {} \;
done
else
echo "Check $USER\'s permissions; cannot change to $CURDIR."
# You can add a mailer here for notification...
exit 1
fi
done


b. Version 2

#!/bin/bash

LOGDIR=/ramsys_data/bmcps/ramsys/
DIRLIST="ALTL AMA5 CAMA FREE NRTL SEGR UTSS"

cd $LOGDIR || {
echo "Check the permissions; can't change to $CURDIR."
# You can add a mailer here for notification...
exit 1
}

for XXX in DIRLIST
do
for PURDIR in `cat SUBDIR.txt`
do
# A log file can be added here [good practice] to list the files to be deleted...
find $XXX/$PURDIR -type f -name "*" -mtime +5 -exec rm {} \;
done
done


If this will be run in cron, e.g. everyday at 0100H, we can make it as:

0 1 * * * ( scriptname_here.sh > /if_wanted/can/store/to/logfile 2>&1 ) &

Or throw it to /dev/null. Whatever suits you.

Scripting 101: Monitor Long-running Autosys Jobs

HELLo World! It's been sometime since I made a "working" script. The last time I remember I created is about 2 years ago [I guess]. I lost interest in programming for a while for some work-related reason and just had this renewed interest a few weeks back. Without much a-doo, here it is: [Sorry 'bout the format, I'm not in the mood to edit it - for now]


#!/bin/bash
### Calls some environment settings to use Autosys
. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN

### Declaration of VARIABLES
declare -i SKIPTIME
declare -i NOMIN
declare -i NOSEC
ERRLOG=/home/$USER/Err.log
E_NOTFOUND=66

### Prompts the user for INPUTS
read -p "Enter the Job Name: " JOBNAME
read -p "Enter the time interval between display (in seconds): " SKIPTIME

### Check to make sure "interval" is integer; don't know what else to do...
if [ $SKIPTIME -le "0" ]
then
echo "Make sure time interval is greater than zero or a number."
exit 1
fi

### This part acts as catch for INVALID Jobs
CHKINVJOB=`autorep -J $JOBNAME`
echo $CHKINVJOB > $ERRLOG

if [ ! -f $ERRLOG ]
then
echo "Error log file does not exist! Check write permissions on the directory."
exit $E_NOTFOUND
exit 1
fi

more $ERRLOG | grep Invalid
if [ $? == "0" ]
then
more $ERRLOG
echo "--- Clearing $ERRLOG ... ---"
cat /dev/null > $ERRLOG
exit 1
fi

### This part checks for the status of the Job
while true
do
autorep -J $JOBNAME | sed -n '5p' | grep -w SU > /dev/null
if [ $? == "0" ]
then
clear
echo "---- Feed Status ----"
echo
echo "***** Job ended SU-ccessfully! *****"
autorep -J $JOBNAME
echo "---- End of Status ----"
exit 0
elif [ $? == "1" ]
then
autorep -J $JOBNAME | sed -n '5p' | egrep -w "ST|AC|RU" > /dev/null
if [ $? == "0" ]
then
if [ "$SKIPTIME" -gt "60" ]
then
NOMIN=`expr $SKIPTIME / 60`
NOSEC=`expr $SKIPTIME % 60`
clear
echo "---- Feed Status ----"
echo "It is still processing... will report in approx. $NOMIN minutes and $NOSEC seconds."
sleep $SKIPTIME
autorep -J $JOBNAME | sed -n '5p'
echo "---- End of Status ----"
else
clear
echo "---- Feed Status ----"
echo "It is still processing... will report in approx. $SKIPTIME seconds."
sleep $SKIPTIME
autorep -J $JOBNAME | sed -n '5p'
echo "---- End of Status ----"
fi
else
clear
echo "---- Feed Status ----"
echo "Check for any ERROR."
autorep -J $JOBNAME
echo "---- End of Status ----"
exit 1
fi
fi
done

exit 0

Mirroring a Root Volume or Disk, and /boot

I slept late last night, or may be late this early morning. Does it make sense? Anyway, when I woke up, I prepared my food. Forget it. Alright, here we go.

I was doing my morning ritual - if got no work - checking my mails, visiting PACLAND, NBA, going thru today's headlines including stock market? Freakin' sick, ayt?! Can't help it esp. in times like these and working in a bank!

Whew! I find it funny writing about this. Another problem is, I was worrying more on the title than the content.

Seriously, here it is.

I was reading RHEL 5 Deployment Guide. Actually, I was continuing. I was considering of skipping the LVM part but decided to go for it anyway. And something, which triggered this blog, caught my attention: The /boot/ partition cannot reside on an LVM volume because the GRUB boot loader cannot read it.

Interesting point. I admit, I was more of an HP-UX LVM guy. Shame on me for the confusion. I begun to ask a former colleague - Che, kudos - to execute some commands for us to verify if it the same with HP-UX. I was so pissed at myself and at the same time wanted to laugh that I was not sure which command to say. Finally, I got me a `setboot`:

[root@server1032:/root]
# setboot
Primary bootpath : 0/1/1/0.0.0
Alternate bootpath : 0/1/1/0.1.0

Autoboot is ON (enabled)
Autosearch is ON (enabled)

Which at that point, doesn't answer the question nor mean anything [yet]?!. So, I Googled it with key words: HP-UX LVM /boot. See? /boot in HP-UX? Another evidence that I wasn't thinking right - since this is an HP-UX issue, I should have gone to ITRC.

Too, late.

... and there it is [as I end up with ITRC]. How can I forget it? It's included in vg00. In fact, there are commands such as mkboot and -B for pvcreate. And please take note of lvlnboot, there is this line that says: Boot.

On the side note, for a bad boot, mirrored disk replacement case, here's how it is done [PA-RISC and Integrity] - this is available on HP's docs:

---
Mirroring the Root Volume on PA-RISC Servers

To set up a mirrored root configuration, you must add a disk to the root volume group, mirror all the root logical volumes onto it, and make it bootable. For this example, the disk is at path 2/0/7.15.0 and has device special files named /dev/rdsk/c2t15d0 and /dev/dsk/c2t15d0.

1. Use the insf command with the -e option to make sure the device files are in place. For example:

# insf -e -H 2/0/7.15.0

You should now have the following device files for this disk:
/dev/dsk/c2t15d0 The entire disk (block access)
/dev/rdsk/c2t15d0 The entire disk (character access)

2. Create a physical volume using pvcreate with the -B option:

# pvcreate -B /dev/rdsk/c2t15d0

3. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t15d0

4. Use the mkboot command to set up the boot area:

# mkboot /dev/rdsk/c2t15d0

5. Use the mkboot command to add an autoboot file to the disk boot area. If you expect to boot from this disk only when you lose quorum, you can use the alternate string “hpux –lq” to disable quorum checking:

# mkboot –a “hpux” /dev/rdsk/c2t15d0

1. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t6d0 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00038 current /dev/vg00/lvol2 00000
00550 current /dev/vg00/lvol3 00000
00583 current /dev/vg00/lvol4 00000
00608 current /dev/vg00/lvol5 00000
00611 current /dev/vg00/lvol6 00000
00923 current /dev/vg00/lvol7 00000
01252 current /dev/vg00/lvol8 00000

In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t15d0

6. Update the root volume group information:

# lvlnboot -R /dev/vg00

7. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

8. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 2/0/7.15.0

9. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t15d0

where l denotes LVM.
---

Mirroring the Root Volume on Integrity Servers

The procedure to mirror the root disk on Integrity servers is similar to the procedure for PA-RISC servers. The difference is that Integrity server boot disks are partitioned; you must set up the partitions, copy utilities to the EFI partition, and use the HP-UX partition device files for LVM commands.

For this example, the disk is at hardware path 0/1/1/0.1.0, with a device special file named /dev/rdsk/c2t1d0.

1. Partition the disk using the idisk command and a partition description file.

a. Create a partition description file. For example:

# vi /tmp/pdf

In this example the partition description file contains:
3
EFI 500MB
HPUX 100%
HPSP 400MB

b. Partition the disk using idisk and the partition description file created in step 1a:

# idisk -f /tmp/pdf -w /dev/rdsk/c2t1d0

To verify that your partitions are correctly laid out, run the following command:

# idisk /dev/rdsk/c2t1d0

2. Use the insf command with the -e option to create the device files for all the partitions. For example:

# insf -e -H 0/1/1/0.1.0

You should now have the following device files for this disk:
/dev/dsk/c2t1d0 The entire disk (block access)
/dev/rdsk/c2t1d0 The entire disk (character access)
/dev/dsk/c2t1d0s1 The EFI partition (block access)
/dev/rdsk/c2t1d0s1 The EFI partition (character access)
/dev/dsk/c2t1d0s2 The HP-UX partition (block access)
/dev/rdsk/c2t1d0s2 The HP-UX partition (character access)
/dev/dsk/c2t1d0s3 The Service partition (block access)
/dev/rdsk/c2t1d0s3 The Service partition (character access)

3. Create a physical volume using pvcreate with the -B option. Be sure to use the device file denoting the HP-UX partition.

# pvcreate -B /dev/rdsk/c2t1d0s2

4. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t1d0s2

5. Use the mkboot command to set up the boot area. Specify the -e and -l options to copy EFI utilities to the EFI partition, and use the device special file for the entire disk:

# mkboot –e –l /dev/rdsk/c2t1d0

6. Update the autoboot file in the EFI partition.

a. Create an AUTO file in the current directory. If you expect to boot from this disk only when you have lost quorum, you can use the alternate string “boot vmunix –lq” to disable quorum checking:

# echo “boot vmunix” > ./AUTO

b. Copy the file from the current directory into the new disk EFI partition. Make sure to use the device file with the s1 suffix:

# efi_cp -d /dev/rdsk/c2t1d0s1 ./AUTO /efi/hpux/auto

7. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t0d0s2 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00010 current /dev/vg00/lvol2 00000
00138 current /dev/vg00/lvol3 00000
00151 current /dev/vg00/lvol4 00000
00158 current /dev/vg00/lvol5 00000
00159 current /dev/vg00/lvol6 00000
00271 current /dev/vg00/lvol7 00000
00408 current /dev/vg00/lvol8 00000

In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t1d0s2

8. Update the root volume group information:

# lvlnboot -R /dev/vg00

9. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

10. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 0/1/1/0.1.0

11. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t1d0s2

where l denotes LVM.
---

My first question has been answered. But, as a former S.A. and UNIX support engineer [for a huge infra], a redundant connection and/or backup is always in the back of our minds. As they say, "He who laughs last, has a backup."

So, the second question is, if /boot is not under the control of LVM, what is a good way to have a redundant connection? I don't have enough experience with this so, I turn to an old friend who has or had. We talked of RAID: software and hardware. For this case, he suggested RAIDS: 1, 5, or 1+0. In case you're server have a RAID controller, it is recommended to go for hardware. This and backup, he said, is enough to get you through.

Basically, 100MB is enough for /boot so, the rest of the disk, in case you decide to use LVM, can be brought under its control. While some others separate / as well for faster recovery process, he continued, for there were cases where the root VG, that contains the default file systems, is not detected. And this is a lengthy process of recovering it. It's a big no-no in critical systems, where downtime is associated with business loss [read: money].

So, I guess, that's it. I'm quite satisfied with the answers, and then some.