Mabuhay

Hello world! This is it. I've always wanted to blog. I don't want no fame but just to let myself heard. No! Just to express myself. So, I don't really care if someone believes in what I'm going to write here nor if ever someone gets interested reading it. My blogs may be a novel-like, a one-liner, it doesn't matter. Still, I'm willing to listen to your views, as long as it justifies mine... Well, enjoy your stay, and I hope you'll learn something new because I just did and sharing it with you.. Welcome!
Showing posts with label scripting. Show all posts
Showing posts with label scripting. Show all posts

Wednesday, November 18, 2009

Perl-icious: Function that reads from a config file

I was asked by my colleague to make a function he'll use for one of our scripts - to make our lives easier which I don't think won't happen unless THEY get rid of the night effin' shift! Anyway, it took me sometime to figure this out (I'm learning ok?!) and then I sought the help of a resident guru BE (his initial you idiot!).

Background:

I was to read the NetAgent configuration file and determine if the server is an MDH or P2PS (this is about RMDS). Another was from the RMDS config file to do the same, which means I got to make two.

Apologies but I cannot post config files here. But I believe you can easily understand how it process things.

Here is the code - as a standalone:

sub inNetAgentConfig {

my $host = shift;
my $conf_file = "/path/to/config.file";

open FH, $conf_file or die "Error: $!\n";
$/ = "ATTRIBUTES"; # set the line delimiter to ATTRIBUTES instead of default value of newline; see special variables
my @array = <FH>;

if ( grep {/$host/} $conf_file ) {
foreach my $line ( grep {/$host/} @array ) {
if ( $line =~ /GROUP=(\w+)/ ) {
return $1;
}
}
} else {
return "Non-existent";
}
}


Another function that I made:


sub inRmdsConfig {

my $host = shift;
my $rmds_file = "/path/to/config.file";

open FILE, $rmds_file or die "Error: $!\n";
while (<FILE>) {
chomp ( $_ );

if ( $_ =~ /^$host\*(\w+)\*serverId.?/ ) {
return $1;
} else {
return "Non-existent";
}
}

Thursday, June 11, 2009

Scripting 101: Modification on the previous topic

Before I modified the original script, I was advised by my colleague to take note of the date especially for the single digits, which was not addressed before. So, the corrections include re-assigning of variables, deletion of unwanted, throwing errors to null, etc.

Here they are:

1. CSV_DATE=`date '+%a %b %d'`
2. LIST_DATE was removed.
3. echo-es were deleted.
4. Errors were thrown to /dev/null; when added to cron, this will be re-directed to /dev/null as well.
5. mail was not working so I used EOF for the body:

/usr/ucb/mail -s "Subject here..." $EMAIL_ADD <<\
EOF

# Whatever the content between EOFs will be evaluated a regular formatted text (including blank lines and spaces) except for variable substitution such as here, which will output the result - the current date.

This will be a regular text.
$(date)

EOF

6. Since LIST_DATE was removed, I used 'find' to locate the most recent files that were modified, as:

for CSV in `find /path/of/files -name "some*.csv" -mtime 0\
2> /dev/null`
do
cp -p $CSV /some/httpd/location/
done

I hope, everything will be ok now after this week's test. It'll be submitted for cron-nization!

Saturday, May 23, 2009

Scripting 101: Shell Script that uses Perl

Good morning (very sleepy now!).. I promised to myself that I will modify the script that we're using to generate a report. I'm not familiar - yet - with the contents but the thing is, we run this manually every night and then copy it to another directory which make it available to the user/s. It's just another basic modification but I added some "twists" a safety pre-caution or stop unwanted operation - well, that's what I wish, at least.

The main script was made by "Someone Else" (There! I ain't saying it's mine.). In time, I'll interpret the Perl part here.


#! /usr/bin/sh

#
# This is a modified version.
# Author: Someone Else
# Modified by: ME
# Renamed: badname_PT.sh
# Date: 23 May, 2009
# Version: 1
#

CSV_DATE=`date | awk '{print $1,$2,$3}'`
E_NOTCD=43
E_OK=0
E_OTHER=45
EMAIL_ADD="name@domain.com"
FILE_DATE=`date '+%d_%m_%y'`
HOSTS="/path/to/host_list_PT"
LIST_DATE=`date | awk '{print $2,$3}'`
SCRIPTDIR="/var/tmp/jf"
export all

# On top of the original script, hour condition was placed to make sure that it runs only after 17:59 daily.
if [ `date +%H` -gt "17" ]
then

cd $SCRIPTDIR || {
/usr/bin/mailx -s "Can't change to $SCRIPTDIR; Please \
check permissions..." $EMAIL_ADD
exit $E_NOTCD
}

# This generates the reports; the heart of the script
for H_LIST in `cat $HOSTS`
do
echo "Extracting bad names from P2PS: $H_LIST"

# No more manual intervention in changing the dates (CSV_DATE was used - from current date)
rsh $H_LIST cat /path/to/some.log* | grep \
"$CSV_DATE" | grep ptrade | perl -n -e \
'if (m/^.* ([0-9]*:[0-9]*:[0-9]*).*Open Failure for (\(.*:.*\)) \
by (\(.*\)) at (\(.*\/net\)).*/) {$_= "$1,$2,$3,$4"; $_ =~ s/[\)|\(]//g;\
print "$_\n";}' > /var/tmp/PT_BadRequests.$H_LIST.csv

sleep 1

echo "Copying /var/tmp/PT_BadRequests.$H_LIST.csv /var/tmp/jf"
cp /var/tmp/PT_BadRequests.$H_LIST.csv /var/tmp/jf
echo "Copy completed for $H_LIST"
done

sleep 3

chmod 666 /var/tmp/jf/PT_*

rm /tmp/PT_BadNames_*
rm /some/httpd/html/PT_BadNames*

tar cvf /tmp/PT_BadNames_$FILE_DATE.tar ./PT*csv
compress /tmp/PT_BadNames*.tar
cp /tmp/PT_BadNames* /some/httpd/html
chmod 666 /some/httpd/html/PT_BadNames*

sleep 2

rm /tmp/PT_BadNames*
rm /var/tmp/jf/PT_BadReq*

# This was added to copy the newly generated CSVs from /var/tmp to /app/httpd/html site
if cd /var/tmp
then
for csv in `ls -l *csv | grep PT_BadRequests | grep "$LIST_DATE" \
| awk '{print $9}'`
do
cp -p $csv /some/httpd/html/Primetrade_BadRequests/
ls -l /some/httpd/html/Primetrade_BadRequests/$csv
done

# On completion, a mail will be sent to intended recipient/s.
/usr/bin/mailx -s "Bad Name PRIMETRADE Report is DONE" $EMAIL_ADD
exit $E_OK

else
/usr/bin/mailx -s "Can't change to /var/tmp; Please check \
permissions..." $EMAIL_ADD
exit $E_NOTCD
fi

fi

echo "Not yet..."
exit $E_OTHER

Also, if this script is run manually, which uses csh, copying each file would be "tedious". So, I made a FOR-loop to do it - very basic but syntax is tricky.

% set LIST_DATE=`date +%b" "%d`
% foreach csv (`ls -l /var/tmp/*csv | grep PT_BadRequests | grep "$LIST_DATE" | awk '{print $9}'`)
? do
? cp -p $csv /some/httpd/html/Primetrade_BadRequests/
? ls -l /some/httpd/html/Primetrade_BadRequests/$csv
? end
%

Sunday, April 19, 2009

Scripting 101: Monitoring Autosys Jobs - Correction in SU_MAILER

Whoa! My prayer was answered! Problem is solved!

Helpful shell scripting guru, from a forum I'm in, was able to address my dilemma for the duplicate entries. I created a confusion which unknowingly had a BIG impact in it's evaluation of regex. Without much-ado, here it is:

sed '/'"$JOBNAME"'/{x;/Y/!{s/^/Y/;h;d;};x;}' JOBLIST.txt

Perfect! I'm just waiting for the explanation. I'm lazy right now to search for these.

Wednesday, April 01, 2009

Scripting 101: Monitoring Autosys Jobs - Changes

When I got to the office today, I started working on the planned changes for the script. 'Though not all, I was able to find a way to implement number 4 from my early list. So here it goes.

Let me establish first the fact that this is primarily designed to address the current setup of jobs that we're monitoring. We already know when it'll be executed so the list is pre-arranged according to time of execution. From which I have considered the changes that I did. And as always, this is a work in progress so bear with me.

First change that I did was to add a couple of lines in the JOB_CHECKER function after the SU_MAILER.

cp -p JOBLIST.txt TEMP.out
sed '1d' TEMP.out > JOBLIST.txt

It deletes job 1 from the JOBLIST.txt as not to waste time when doing a FOR-loop for jobs already marked as SU. It's kinda lame really but am looking for better way to cleanup this mess.

The second one is changing the condition in JOB_CHECKER function.

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]

was changed to [and DONE_JOB is removed]:

if grep -w $ONQUEUE_JOB EXCLUDEJOB.txt

Third is on WHILE-loop. The script exits when there's nothing left [to read] in the JOBLIST.txt. Hence, COUNTER is no longer needed.

while [ "$NO_JOBS" -gt "0" ]
...

It's still consuming a lot of CPU and I haven't figure this one out yet. To allow "others", after a FOR-loop, I put the script into sleep.

And lastly, a safety net was added in the SU_MAILER function. It'll erase duplicate entries from the exclusion list.

SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

awk '!x[$0]++' EXCLUDEJOB.txt > EXCLUDEJOB.txt

I hope I'm on track on improving my scripting skills; always looking forward for a new challenge.

********
This part was added on 15 April, 2009 at 21:33 SST.

The variable $NO_JOBS was removed. Instead, its value was placed in the WHILE-loop. It was late when I found out that it is NOT being updated. Also, there are times when some of the jobs listed may be invalid for one reason or another. With this, condition was added as well. So the code will be:

while [ `cat JOBLIST.txt | wc -l` -gt "0" ]
do
for JOBNAME in `cat JOBLIST.txt`
do

if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep "Invalid Job Name" 2>&1 > /dev/null
then
cp -p JOBLIST.txt VALJOB.out
sed -e "\|^$JOBNAME\$|d" VALJOB.out > JOBLIST.txt
continue
fi

...
done

exit $E_DONE

SU_MAILER () is still in the works. I still can't find how to delete duplicate entries while leaving one of them. The "awk" part that previously used was scrapped. It's NOT up to the job. Of course, it's my fault. ;)

I really hope I'll have the flame to continue on this for I started on my new job and is being "forced" to take on Perl which seems great. Now I know that it ain't a scripting language but rather on the grey area between interpreted and compiled. Still a long wayyyyyy. See'ya round.

Scripting 101: Monitoring Autosys Jobs - Evolved Form

Finally, here it is! I have this tested last night - on a PROD server [Linux only for some commands might NOT run on Solaris] - and it went well. After making some adjustments, it did behave as expected. Wow, I can't believe it. After a long while, I did it. Not of course without the valuable insights from my mentors - CJ and Mors - who gave their all [I hope!?]. I am a student and beginner in scripting. As for now, I'm still analyzing the logs to maximize performance for I have seen that it wastes so much time in loops. So anyone out there, please help me!

Here are some of what I've observed and future plans:
1. Run via cron; OR run it via "screen"
2. High CPU utilization
3. Integrate mutt in the script [not sure where but I'll try; it's a bit cool, ain't it?]
4. Instead of having an exclusion file, after a job is done, it will remove from the JOBLIST.txt and exit when nothing to read.

**********
#! /bin/bash

# Calling AUTOSYS environment
. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN

# Definition and initialization of VARIABLES.
E_NOTFOUND=66
E_MISSING=43
E_DONE=0
EMAIL_ADD=`whoami`
COUNTER=0
NO_JOBS=`cat JOBLIST.txt | wc -l`


# Initialization of EXCLUSION file; this will be used to exclude successful job
> EXCLUDEJOB.txt
/bin/touch EXCLUDEJOB.txt
if [ ! -w EXCLUDEJOB.txt ]
then
/bin/echo "EXCLUDEJOB.txt file does not exist and/or not writeable! Check write permissions on the directory." | /bin/mail -s "EXCLUDEJOB.txt Missing" $EMAIL_ADD
exit $E_NOTFOUND
fi


# This is a mailer function for successful jobs; once the job ended successfully, it will write to EXCLUDEJOB.txt before sending a mail
SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

/bin/mail -s "FOS Job Monitor Update" $EMAIL_ADD >>EOF
****** Feed Status ******

Job ended SU-ccessfully!
$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)

***** End of Status *****
EOF
} # End of SU_MAILER function


FA_MAILER() {
/bin/mail -s "FOS Job Monitor Alert" $EMAIL_ADD >> EOF
***** PLEASE check Job Status *****

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)
$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -q -J $JOBNAME)

***** End of Status ****
EOF
} # End of FA_MAILER function


JOB_CHECKER() {
DONE_JOB=`grep $JOBNAME EXCLUDEJOB.txt`
ONQUEUE_JOB=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}'`

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]
then
continue
else
JOB_TIMESTAMP=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $3}'`

if [ "$DATE_STAMP" = "$JOB_DATE" ] && [ "$TIME_STAMP" -ge "$JOB_TIMESTAMP" ]
then
if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | grep -w SU 2>&1 > /dev/null
then

SU_MAILER
COUNTER=$[$COUNTER+1]

elif /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | egrep -w "ST|AC|RU" 2>&1 > /dev/null
then
break
else

FA_MAILER
COUNTER=$[$COUNTER+1]

fi
fi
fi
}
# End of JOB_CHECKER function

# This is basically the main function of the script; it will continue to run until all jobs listed in the JOBLIST.txt are successful

while [ "$COUNTER" -lt "$NO_JOBS" ]
do
for JOBNAME in `cat JOBLIST.txt`
do
JOB_TZ=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME -q | grep timezone | awk '{print $2}' | cut -d/ -f2`

if [ -z "$JOB_TZ" ]
then
/bin/echo "Timezone is NOT defined in $JOBNAME properties. Please check." | /bin/mail -s "Missing TIMEZONE" $EMAIL_ADD
continue
else

JOB_DATE=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{print $2}'`

case $JOB_TZ in
London)
DATE_STAMP=`TZ=GMT date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Zurich)
DATE_STAMP=`TZ=GMT-1 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-1 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Eastern)
DATE_STAMP=`TZ=GMT+5 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT+5 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Singapore|HongKong)
DATE_STAMP=`TZ=GMT-8 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-8 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Tokyo)
DATE_STAMP=`TZ=GMT-9 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-9 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Sydney)
DATE_STAMP=`TZ=GMT-11 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-11 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

*)
/bin/echo "What else could go wrong? Notify the script owner... Stressful." | /bin/mail -s "FOlSe Alert. Something is missing..." $EMAIL_ADD
exit $E_MISSING;;
esac
fi

JOB_CHECKER

done # End of FOR-loop

# Sleep will be added here for about 5 minutes
sleep 300

done
# End of WHILE-loop

exit $E_DONE

Thursday, January 29, 2009

Scripting 101: Purging Files

Ei-lo! 'Zup? I'm beginning to enjoy this I guess: scripting. Unexpected, I got a ping from a former colleague, asking from a noobs [read: me] regarding scripting. Without blinking, I accepted the challenge. I was asked to check on the script she made for purging files. After checking on the requirements, given, and asking for advise from a scripting guru [Morsgang Freeman], below are the revised forms. Btw, the first one I made really sucks [I hope these won't :(]. As of writing, these are not tested, not 'til tomorrow when she receives the email/script.

1. First is to create a file, say SUBDIR.txt with the ff. contents:


MERGE/REJECTS/ARCHIVE

MERGE/REJECTS/ARCHIVERAW

MERGE/FILTERED/ARCHIVE

MERGE/FILTERED/ARCHIVERAW

INSERT/REJECTS/INSERTED

INSERT/FILTERED/INSERTED

FTP/FTP_ARCHIVE


2. I made two variations of the script:
a. Version 1

#!/bin/bash

export LOGDIR=/ramsys_data/bmcps/ramsys/
export DIRLIST="ALTL AMA5 CAMA FREE NRTL SEGR UTSS"

for XXX in DIRLIST
do
cd $LOGDIR
if [ "$PWD" = "$LOGDIR" ]
then
for PURDIR in `cat SUBDIR.txt`
do
# A log file can be added here [good practice] to list the files to be deleted...
find $XXX/$PURDIR -type f -name "*" -mtime +5 -exec rm {} \;
done
else
echo "Check $USER\'s permissions; cannot change to $CURDIR."
# You can add a mailer here for notification...
exit 1
fi
done


b. Version 2

#!/bin/bash

LOGDIR=/ramsys_data/bmcps/ramsys/
DIRLIST="ALTL AMA5 CAMA FREE NRTL SEGR UTSS"

cd $LOGDIR || {
echo "Check the permissions; can't change to $CURDIR."
# You can add a mailer here for notification...
exit 1
}

for XXX in DIRLIST
do
for PURDIR in `cat SUBDIR.txt`
do
# A log file can be added here [good practice] to list the files to be deleted...
find $XXX/$PURDIR -type f -name "*" -mtime +5 -exec rm {} \;
done
done


If this will be run in cron, e.g. everyday at 0100H, we can make it as:

0 1 * * * ( scriptname_here.sh > /if_wanted/can/store/to/logfile 2>&1 ) &

Or throw it to /dev/null. Whatever suits you.

Wednesday, January 28, 2009

Scripting 101: Monitor Long-running Autosys Jobs

HELLo World! It's been sometime since I made a "working" script. The last time I remember I created is about 2 years ago [I guess]. I lost interest in programming for a while for some work-related reason and just had this renewed interest a few weeks back. Without much a-doo, here it is: [Sorry 'bout the format, I'm not in the mood to edit it - for now]


#!/bin/bash
### Calls some environment settings to use Autosys

. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN


### Declaration of VARIABLES

declare -i SKIPTIME
declare -i NOMIN
declare -i NOSEC

ERRLOG=/home/$USER/Err.log

E_NOTFOUND=66


### Prompts the user for INPUTS

read -p "Enter the Job Name: " JOBNAME

read -p "Enter the time interval between display (in seconds): " SKIPTIME


### Check to make sure "interval" is integer; don't know what else to
do...
if [ $SKIPTIME -le "0" ]
then
echo "Make sure time interval is greater than zero or a number."
exit 1
fi

### This part acts as catch for INVALID Jobs
CHKINVJOB=`autorep -J $JOBNAME`
echo $CHKINVJOB > $ERRLOG

if [ ! -f $ERRLOG ]
then
echo "Error log file does not exist! Check write permissions on the directory."
exit $E_NOTFOUND
exit 1
fi

more $ERRLOG | grep Invalid
if [ $? == "0" ]
then
more $ERRLOG
echo "--- Clearing $ERRLOG ... ---"
cat /dev/null > $ERRLOG
exit 1
fi

### This part checks for the status of the Job
while true
do
autorep -J $JOBNAME | sed -n '5p' | grep -w SU > /dev/null
if [ $? == "0" ]
then
clear
echo "---- Feed Status ----"
echo
echo "***** Job ended SU-ccessfully! *****"
autorep -J $JOBNAME
echo "---- End of Status ----"
exit 0
elif [ $? == "1" ]
then
autorep -J $JOBNAME | sed -n '5p' | egrep -w "ST|AC|RU" > /dev/null
if [ $? == "0" ]
then
if [ "$SKIPTIME" -gt "60" ]
then
NOMIN=`expr $SKIPTIME / 60`
NOSEC=`expr $SKIPTIME % 60`
clear
echo "---- Feed Status ----"
echo "It is still processing... will report in approx. $NOMIN minutes and $NOSEC seconds."
sleep $SKIPTIME
autorep -J $JOBNAME | sed -n '5p'
echo "---- End of Status ----"
else
clear
echo "---- Feed Status ----"
echo "It is still processing... will report in approx. $SKIPTIME seconds."
sleep $SKIPTIME
autorep -J $JOBNAME | sed -n '5p'
echo "---- End of Status ----"
fi
else
clear
echo "---- Feed Status ----"
echo "Check for any ERROR."
autorep -J $JOBNAME
echo "---- End of Status ----"
exit 1
fi
fi
done

exit 0

World Clock