Mabuhay

Hello world! This is it. I've always wanted to blog. I don't want no fame but just to let myself heard. No! Just to express myself. So, I don't really care if someone believes in what I'm going to write here nor if ever someone gets interested reading it. My blogs may be a novel-like, a one-liner, it doesn't matter. Still, I'm willing to listen to your views, as long as it justifies mine... Well, enjoy your stay, and I hope you'll learn something new because I just did and sharing it with you.. Welcome!
Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Wednesday, April 01, 2009

Scripting 101: Monitoring Autosys Jobs - Evolved Form

Finally, here it is! I have this tested last night - on a PROD server [Linux only for some commands might NOT run on Solaris] - and it went well. After making some adjustments, it did behave as expected. Wow, I can't believe it. After a long while, I did it. Not of course without the valuable insights from my mentors - CJ and Mors - who gave their all [I hope!?]. I am a student and beginner in scripting. As for now, I'm still analyzing the logs to maximize performance for I have seen that it wastes so much time in loops. So anyone out there, please help me!

Here are some of what I've observed and future plans:
1. Run via cron; OR run it via "screen"
2. High CPU utilization
3. Integrate mutt in the script [not sure where but I'll try; it's a bit cool, ain't it?]
4. Instead of having an exclusion file, after a job is done, it will remove from the JOBLIST.txt and exit when nothing to read.

**********
#! /bin/bash

# Calling AUTOSYS environment
. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN

# Definition and initialization of VARIABLES.
E_NOTFOUND=66
E_MISSING=43
E_DONE=0
EMAIL_ADD=`whoami`
COUNTER=0
NO_JOBS=`cat JOBLIST.txt | wc -l`


# Initialization of EXCLUSION file; this will be used to exclude successful job
> EXCLUDEJOB.txt
/bin/touch EXCLUDEJOB.txt
if [ ! -w EXCLUDEJOB.txt ]
then
/bin/echo "EXCLUDEJOB.txt file does not exist and/or not writeable! Check write permissions on the directory." | /bin/mail -s "EXCLUDEJOB.txt Missing" $EMAIL_ADD
exit $E_NOTFOUND
fi


# This is a mailer function for successful jobs; once the job ended successfully, it will write to EXCLUDEJOB.txt before sending a mail
SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

/bin/mail -s "FOS Job Monitor Update" $EMAIL_ADD >>EOF
****** Feed Status ******

Job ended SU-ccessfully!
$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)

***** End of Status *****
EOF
} # End of SU_MAILER function


FA_MAILER() {
/bin/mail -s "FOS Job Monitor Alert" $EMAIL_ADD >> EOF
***** PLEASE check Job Status *****

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)
$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -q -J $JOBNAME)

***** End of Status ****
EOF
} # End of FA_MAILER function


JOB_CHECKER() {
DONE_JOB=`grep $JOBNAME EXCLUDEJOB.txt`
ONQUEUE_JOB=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}'`

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]
then
continue
else
JOB_TIMESTAMP=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $3}'`

if [ "$DATE_STAMP" = "$JOB_DATE" ] && [ "$TIME_STAMP" -ge "$JOB_TIMESTAMP" ]
then
if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | grep -w SU 2>&1 > /dev/null
then

SU_MAILER
COUNTER=$[$COUNTER+1]

elif /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | egrep -w "ST|AC|RU" 2>&1 > /dev/null
then
break
else

FA_MAILER
COUNTER=$[$COUNTER+1]

fi
fi
fi
}
# End of JOB_CHECKER function

# This is basically the main function of the script; it will continue to run until all jobs listed in the JOBLIST.txt are successful

while [ "$COUNTER" -lt "$NO_JOBS" ]
do
for JOBNAME in `cat JOBLIST.txt`
do
JOB_TZ=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME -q | grep timezone | awk '{print $2}' | cut -d/ -f2`

if [ -z "$JOB_TZ" ]
then
/bin/echo "Timezone is NOT defined in $JOBNAME properties. Please check." | /bin/mail -s "Missing TIMEZONE" $EMAIL_ADD
continue
else

JOB_DATE=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{print $2}'`

case $JOB_TZ in
London)
DATE_STAMP=`TZ=GMT date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Zurich)
DATE_STAMP=`TZ=GMT-1 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-1 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Eastern)
DATE_STAMP=`TZ=GMT+5 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT+5 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Singapore|HongKong)
DATE_STAMP=`TZ=GMT-8 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-8 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Tokyo)
DATE_STAMP=`TZ=GMT-9 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-9 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Sydney)
DATE_STAMP=`TZ=GMT-11 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-11 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

*)
/bin/echo "What else could go wrong? Notify the script owner... Stressful." | /bin/mail -s "FOlSe Alert. Something is missing..." $EMAIL_ADD
exit $E_MISSING;;
esac
fi

JOB_CHECKER

done # End of FOR-loop

# Sleep will be added here for about 5 minutes
sleep 300

done
# End of WHILE-loop

exit $E_DONE

Sunday, January 04, 2009

Mirroring a Root Volume or Disk, and /boot

I slept late last night, or may be late this early morning. Does it make sense? Anyway, when I woke up, I prepared my food. Forget it. Alright, here we go.

I was doing my morning ritual - if got no work - checking my mails, visiting PACLAND, NBA, going thru today's headlines including stock market? Freakin' sick, ayt?! Can't help it esp. in times like these and working in a bank!

Whew! I find it funny writing about this. Another problem is, I was worrying more on the title than the content.

Seriously, here it is.

I was reading RHEL 5 Deployment Guide. Actually, I was continuing. I was considering of skipping the LVM part but decided to go for it anyway. And something, which triggered this blog, caught my attention: The /boot/ partition cannot reside on an LVM volume because the GRUB boot loader cannot read it.

Interesting point. I admit, I was more of an HP-UX LVM guy. Shame on me for the confusion. I begun to ask a former colleague - Che, kudos - to execute some commands for us to verify if it the same with HP-UX. I was so pissed at myself and at the same time wanted to laugh that I was not sure which command to say. Finally, I got me a `setboot`:

[root@server1032:/root]
# setboot
Primary bootpath : 0/1/1/0.0.0

Alternate bootpath : 0/1/1/0.1.0


Autoboot is ON (enabled)

Autosearch is ON (enabled)


Which at that point, doesn't answer the question nor mean anything [yet]?!. So, I Googled it with key words: HP-UX LVM /boot. See? /boot in HP-UX? Another evidence that I wasn't thinking right - since this is an HP-UX issue, I should have gone to ITRC.

Too, late.

... and there it is [as I end up with ITRC]. How can I forget it? It's included in vg00. In fact, there are commands such as mkboot and -B for pvcreate. And please take note of lvlnboot, there is this line that says: Boot.

On the side note, for a bad boot, mirrored disk replacement case, here's how it is done [PA-RISC and Integrity] - this is available on HP's docs:

---
Mirroring the Root Volume on PA-RISC Servers

To set up a mirrored root configuration, you must add a disk to the root volume group, mirror all the root logical volumes onto it, and make it bootable. For this example, the disk is at path 2/0/7.15.0 and has device special files named /dev/rdsk/c2t15d0 and /dev/dsk/c2t15d0.

1. Use the insf command with the -e option to make sure the device files are in place. For example:

# insf -e -H 2/0/7.15.0

You should now have the following device files for this disk:
/dev/dsk/c2t15d0 The entire disk (block access)
/dev/rdsk/c2t15d0 The entire disk (character access)

2. Create a physical volume using pvcreate with the -B option:

# pvcreate -B /dev/rdsk/c2t15d0

3. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t15d0

4. Use the mkboot command to set up the boot area:

# mkboot /dev/rdsk/c2t15d0

5. Use the mkboot command to add an autoboot file to the disk boot area. If you expect to boot from this disk only when you lose quorum, you can use the alternate string “hpux –lq” to disable quorum checking:

# mkboot –a “hpux” /dev/rdsk/c2t15d0

1. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t6d0 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00038 current /dev/vg00/lvol2 00000
00550 current /dev/vg00/lvol3 00000
00583 current /dev/vg00/lvol4 00000
00608 current /dev/vg00/lvol5 00000
00611 current /dev/vg00/lvol6 00000
00923 current /dev/vg00/lvol7 00000
01252 current /dev/vg00/lvol8 00000


In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t15d0

# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t15d0


6. Update the root volume group information:

# lvlnboot -R /dev/vg00

7. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

8. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 2/0/7.15.0

9. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t15d0


where l denotes LVM.
---

Mirroring the Root Volume on Integrity Servers

The procedure to mirror the root disk on Integrity servers is similar to the procedure for PA-RISC servers. The difference is that Integrity server boot disks are partitioned; you must set up the partitions, copy utilities to the EFI partition, and use the HP-UX partition device files for LVM commands.

For this example, the disk is at hardware path 0/1/1/0.1.0, with a device special file named /dev/rdsk/c2t1d0.

1. Partition the disk using the idisk command and a partition description file.

a. Create a partition description file. For example:

# vi /tmp/pdf

In this example the partition description file contains:
3
EFI 500MB
HPUX 100%
HPSP 400MB

b. Partition the disk using idisk and the partition description file created in step 1a:

# idisk -f /tmp/pdf -w /dev/rdsk/c2t1d0

To verify that your partitions are correctly laid out, run the following command:

# idisk /dev/rdsk/c2t1d0

2. Use the insf command with the -e option to create the device files for all the partitions. For example:

# insf -e -H 0/1/1/0.1.0

You should now have the following device files for this disk:
/dev/dsk/c2t1d0 The entire disk (block access)
/dev/rdsk/c2t1d0 The entire disk (character access)
/dev/dsk/c2t1d0s1 The EFI partition (block access)
/dev/rdsk/c2t1d0s1 The EFI partition (character access)
/dev/dsk/c2t1d0s2 The HP-UX partition (block access)
/dev/rdsk/c2t1d0s2 The HP-UX partition (character access)
/dev/dsk/c2t1d0s3 The Service partition (block access)
/dev/rdsk/c2t1d0s3 The Service partition (character access)

3. Create a physical volume using pvcreate with the -B option. Be sure to use the device file denoting the HP-UX partition.

# pvcreate -B /dev/rdsk/c2t1d0s2

4. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t1d0s2

5. Use the mkboot command to set up the boot area. Specify the -e and -l options to copy EFI utilities to the EFI partition, and use the device special file for the entire disk:

# mkboot –e –l /dev/rdsk/c2t1d0

6. Update the autoboot file in the EFI partition.

a. Create an AUTO file in the current directory. If you expect to boot from this disk only when you have lost quorum, you can use the alternate string “boot vmunix –lq” to disable quorum checking:

# echo “boot vmunix” > ./AUTO

b. Copy the file from the current directory into the new disk EFI partition. Make sure to use the device file with the s1 suffix:

# efi_cp -d /dev/rdsk/c2t1d0s1 ./AUTO /efi/hpux/auto

7. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t0d0s2 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00010 current /dev/vg00/lvol2 00000
00138 current /dev/vg00/lvol3 00000
00151 current /dev/vg00/lvol4 00000
00158 current /dev/vg00/lvol5 00000
00159 current /dev/vg00/lvol6 00000
00271 current /dev/vg00/lvol7 00000
00408 current /dev/vg00/lvol8 00000


In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t1d0s2

# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t1d0s2


8. Update the root volume group information:

# lvlnboot -R /dev/vg00

9. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

10. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 0/1/1/0.1.0

11. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t1d0s2


where l denotes LVM.
---

My first question has been answered. But, as a former S.A. and UNIX support engineer [for a huge infra], a redundant connection and/or backup is always in the back of our minds. As they say, "He who laughs last, has a backup."

So, the second question is, if /boot is not under the control of LVM, what is a good way to have a redundant connection? I don't have enough experience with this so, I turn to an old friend who has or had. We talked of RAID: software and hardware. For this case, he suggested RAIDS: 1, 5, or 1+0. In case you're server have a RAID controller, it is recommended to go for hardware. This and backup, he said, is enough to get you through.

Basically, 100MB is enough for /boot so, the rest of the disk, in case you decide to use LVM, can be brought under its control. While some others separate / as well for faster recovery process, he continued, for there were cases where the root VG, that contains the default file systems, is not detected. And this is a lengthy process of recovering it. It's a big no-no in critical systems, where downtime is associated with business loss [read: money].

So, I guess, that's it. I'm quite satisfied with the answers, and then some.

Sunday, April 27, 2008

[SOLVED] Ubuntu 8.04 LTS - Hardy Heron has landed[!] and my NIC disappeared!

At last, it's here! Ok, it was released days ago but at least, the wait is over. :)

It took me about 18 hours to complete the upgrade online via SmartBro[ken]! That's correct, 18 hours. I'm tired of their service. "Konting tiis na lang, papa-putol ko rin 'yung service".

After 18 excruciating hours, my Ubuntu is now a Hardy Heron! Bwahahaha! Hmmm, there are 5 packages that we're not upgraded because of dependency issues, I'll try to fix it later. I rebooted to M$ XP [I'm having a dual-boot,. Need to access my work's email..] and boom! The network's gone! I mean, my NIC is not recognized both on M$ and Ubuntu. Oh sh*t! What happened? I'll check the forums bit later if the upgrade affected my hardware. I hope not [fingers crossed].

But for those who want to try Ubuntu, get it here - http://www.ubuntu.com/getubuntu/download.

Don't let my experience here shun you from trying Linux. It's an excellent OS. And it's free. If you want, I'd be happy to lend a hand. Going back, it could be a hardware issue either. This is just an isolated case! Nothing to worry. I'll update this one later, too! Wish me luck, I need them! Bye!

----------------------------------------------------------

Hello there, I decided to add my update here rather than create a new [notice the line above?].

Hoooooooooooooo! Finally, I got my NIC working! I read through the forums and I was reminded of Troubleshooting 101. Before blaming the OS or software for all the stupidity we have done, please check the hardware [issue] first. Just like in my case, it was a slot problem. I just moved my NIC to the other slot, and whoala, it's working again! Now that I'm back, it's time to fix the remaining packages.

Here are the error logs I got:


Setting up exim4-config (4.69-2) ...
/usr/sbin/dpkg-statoverride: non-existing group Debian-exim

Usage: dpkg-statoverride [ ...]

Commands:
--add
add a new entry into the database.
--remove remove file from the database.
--list [] list current overrides in the database.

Options:
--admindir set the directory with the statoverride file.
--update immediately update file permissions.
--force force an action even if a sanity check fails.
--quiet quiet operation, minimal output.
--help show this help message.
--version show the version.
dpkg: error processing exim4-config (--configure):
subprocess post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of exim4-base:
exim4-base depends on exim4-config (>= 4.30) | exim4-config-2; however:
Package exim4-config is not configured yet.
Package exim4-config-2 is not installed.
Package exim4-config which provides exim4-config-2 is not configured yet.
dpkg: error processing exim4-base (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of exim4-daemon-light:
exim4-daemon-light depends on exim4-base (>= 4.69); however:
Package exim4-base is not configured yet.
dpkg: error processing exim4-daemon-light (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of exim4:
exim4 depends on exim4-base (>= 4.69); however:
Package exim4-base is not configured yet.
exim4 depends on exim4-daemon-light | exim4-daemon-heavy | exim4-daemon-custom; however:
Package exim4-daemon-light is not configured yet.
Package exim4-daemon-heavy is not installed.
Package exim4-daemon-custom is not installed.
dpkg: error processing exim4 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of rkhunter:
rkhunter depends on exim4 | postfix | sendmail | mail-transport-agent; however:
Package exim4 is not configured yet.
Package postfix is not installed.
Package sendmail is not installed.
Package mail-transport-agent is not installed.
Package exim4-daemon-light which provides mail-transport-agent is not configured yet.
dpkg: error processing rkhunter (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
exim4-config
exim4-base
exim4-daemon-light
exim4
rkhunter
E: Sub-process /usr/bin/dpkg returned an error code (1)


At first, I tried to build the dependencies.


root@rot:/house# apt-get build-dep exim4-config
...


'Though this might have corrected the dependencies, I still got the same error. How stupid of me to look ever simple things??? I may be better off joining the army or the police. I just shoot without asking. I checked again. Hmmmm, let's see... It seems the error is about the group Debian-exim, right?!?
Haiz!

In the famous words of my Indian instructor,
"Just do one thing..." well, I did two:


user1@rot:~$ sudo groupadd Debian-exim
user1@rot:~$ sudo /usr/sbin/dpkg-statoverride --add Debian-exim Debian-exim 640 /var/spool/exim4/
stripping trailing /
user1@rot:~$


And then ran again the update. and the rest is history...


root@rot:/house# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
5 not fully installed or removed.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]? y
Setting up exim4-config (4.69-2) ...

Setting up exim4-base (4.69-2) ...
Installing new version of config file /etc/cron.daily/exim4-base ...
Installing new version of config file /etc/init.d/exim4 ...

Setting up exim4-daemon-light (4.69-2) ...
* Starting MTA [ OK ]

Setting up exim4 (4.69-2) ...

Setting up rkhunter (1.3.0-3) ...
Installing new version of config file /etc/rkhunter.conf ...
Installing new version of config file /etc/cron.daily/rkhunter ...
Installing new version of config file /etc/cron.weekly/rkhunter ...
Updating the file properties database:
[ Rootkit Hunter version 1.3.0 ]
File updated: searched for 151 files, found 125

root@rot:/house#

World Clock