Scripting 101: Monitoring Autosys Jobs - Correction in SU_MAILER

Whoa! My prayer was answered! Problem is solved!

Helpful shell scripting guru, from a forum I'm in, was able to address my dilemma for the duplicate entries. I created a confusion which unknowingly had a BIG impact in it's evaluation of regex. Without much-ado, here it is:

sed '/'"$JOBNAME"'/{x;/Y/!{s/^/Y/;h;d;};x;}' JOBLIST.txt

Perfect! I'm just waiting for the explanation. I'm lazy right now to search for these.

Scripting 101: Monitoring Autosys Jobs - Changes

When I got to the office today, I started working on the planned changes for the script. 'Though not all, I was able to find a way to implement number 4 from my early list. So here it goes.

Let me establish first the fact that this is primarily designed to address the current setup of jobs that we're monitoring. We already know when it'll be executed so the list is pre-arranged according to time of execution. From which I have considered the changes that I did. And as always, this is a work in progress so bear with me.

First change that I did was to add a couple of lines in the JOB_CHECKER function after the SU_MAILER.

cp -p JOBLIST.txt TEMP.out

sed '1d' TEMP.out > JOBLIST.txt

It deletes job 1 from the JOBLIST.txt as not to waste time when doing a FOR-loop for jobs already marked as SU. It's kinda lame really but am looking for better way to cleanup this mess.

The second one is changing the condition in JOB_CHECKER function.

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]

was changed to [and DONE_JOB is removed]:

if grep -w $ONQUEUE_JOB EXCLUDEJOB.txt

Third is on WHILE-loop. The script exits when there's nothing left [to read] in the JOBLIST.txt. Hence, COUNTER is no longer needed.

while [ "$NO_JOBS" -gt "0" ]

...

It's still consuming a lot of CPU and I haven't figure this one out yet. To allow "others", after a FOR-loop, I put the script into sleep.

And lastly, a safety net was added in the SU_MAILER function. It'll erase duplicate entries from the exclusion list.

SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

awk '!x[$0]++' EXCLUDEJOB.txt > EXCLUDEJOB.txt

I hope I'm on track on improving my scripting skills; always looking forward for a new challenge.

********

This part was added on 15 April, 2009 at 21:33 SST.

The variable $NO_JOBS was removed. Instead, its value was placed in the WHILE-loop. It was late when I found out that it is NOT being updated. Also, there are times when some of the jobs listed may be invalid for one reason or another. With this, condition was added as well. So the code will be:

while [ `cat JOBLIST.txt | wc -l` -gt "0" ]

do

for JOBNAME in `cat JOBLIST.txt`

do

if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep "Invalid Job Name" 2>&1 > /dev/null

then

cp -p JOBLIST.txt VALJOB.out

sed -e "\|^$JOBNAME\$|d" VALJOB.out > JOBLIST.txt

continue

fi

...

done

exit $E_DONE

SU_MAILER () is still in the works. I still can't find how to delete duplicate entries while leaving one of them. The "awk" part that previously used was scrapped. It's NOT up to the job. Of course, it's my fault. ;)

I really hope I'll have the flame to continue on this for I started on my new job and is being "forced" to take on Perl which seems great. Now I know that it ain't a scripting language but rather on the grey area between interpreted and compiled. Still a long wayyyyyy. See'ya round.

Scripting 101: Monitoring Autosys Jobs - Evolved Form

Finally, here it is! I have this tested last night - on a PROD server [Linux only for some commands might NOT run on Solaris] - and it went well. After making some adjustments, it did behave as expected. Wow, I can't believe it. After a long while, I did it. Not of course without the valuable insights from my mentors - CJ and Mors - who gave their all [I hope!?]. I am a student and beginner in scripting. As for now, I'm still analyzing the logs to maximize performance for I have seen that it wastes so much time in loops. So anyone out there, please help me!

Here are some of what I've observed and future plans:

1. Run via cron; OR run it via "screen"

2. High CPU utilization

3. Integrate mutt in the script [not sure where but I'll try; it's a bit cool, ain't it?]

4. Instead of having an exclusion file, after a job is done, it will remove from the JOBLIST.txt and exit when nothing to read.

**********

#! /bin/bash

# Calling AUTOSYS environment
. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN

# Definition and initialization of VARIABLES.
E_NOTFOUND=66
E_MISSING=43
E_DONE=0
EMAIL_ADD=`whoami`
COUNTER=0
NO_JOBS=`cat JOBLIST.txt | wc -l`


# Initialization of EXCLUSION file; this will be used to exclude successful job

> EXCLUDEJOB.txt
/bin/touch EXCLUDEJOB.txt
if [ ! -w EXCLUDEJOB.txt ]
then
/bin/echo "EXCLUDEJOB.txt file does not exist and/or not writeable! Check write permissions on the directory." | /bin/mail -s "EXCLUDEJOB.txt Missing" $EMAIL_ADD
exit $E_NOTFOUND
fi


# This is a mailer function for successful jobs; once the job ended successfully, it will write to EXCLUDEJOB.txt before sending a mail
SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

/bin/mail -s "FOS Job Monitor Update" $EMAIL_ADD >>EOF

****** Feed Status ******

Job ended SU-ccessfully!

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)

***** End of Status *****

EOF

} # End of SU_MAILER function

FA_MAILER() {

/bin/mail -s "FOS Job Monitor Alert" $EMAIL_ADD >> EOF

***** PLEASE check Job Status *****

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -q -J $JOBNAME)

***** End of Status ****

EOF

} # End of FA_MAILER function

JOB_CHECKER() {
DONE_JOB=`grep $JOBNAME EXCLUDEJOB.txt`
ONQUEUE_JOB=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}'`

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]
then
continue
else
JOB_TIMESTAMP=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $3}'`

if [ "$DATE_STAMP" = "$JOB_DATE" ] && [ "$TIME_STAMP" -ge "$JOB_TIMESTAMP" ]
then
if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | grep -w SU 2>&1 > /dev/null
then

SU_MAILER
COUNTER=$[$COUNTER+1]

elif /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | egrep -w "ST|AC|RU" 2>&1 > /dev/null
then
break
else

FA_MAILER
COUNTER=$[$COUNTER+1]

fi
fi
fi
} # End of JOB_CHECKER function

# This is basically the main function of the script; it will continue to run until all jobs listed in the JOBLIST.txt are successful
while [ "$COUNTER" -lt "$NO_JOBS" ]
do
for JOBNAME in `cat JOBLIST.txt`
do
JOB_TZ=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME -q | grep timezone | awk '{print $2}' | cut -d/ -f2`

if [ -z "$JOB_TZ" ]
then
/bin/echo "Timezone is NOT defined in $JOBNAME properties. Please check." | /bin/mail -s "Missing TIMEZONE" $EMAIL_ADD
continue
else

JOB_DATE=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{print $2}'`

case $JOB_TZ in
London)
DATE_STAMP=`TZ=GMT date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Zurich)
DATE_STAMP=`TZ=GMT-1 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-1 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Eastern)
DATE_STAMP=`TZ=GMT+5 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT+5 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Singapore|HongKong)
DATE_STAMP=`TZ=GMT-8 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-8 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Tokyo)
DATE_STAMP=`TZ=GMT-9 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-9 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Sydney)
DATE_STAMP=`TZ=GMT-11 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-11 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

*)
/bin/echo "What else could go wrong? Notify the script owner... Stressful." | /bin/mail -s "FOlSe Alert. Something is missing..." $EMAIL_ADD
exit $E_MISSING;;
esac
fi

JOB_CHECKER

done # End of FOR-loop

# Sleep will be added here for about 5 minutes

sleep 300

done # End of WHILE-loop

exit $E_DONE