UNIX/Linux Tips [and Tricks?]

Market Data Notes (under construction!)

2011-02-18T19:12:00.010+08:00

Hey chaps! 'Zup? It's been a while since I `touch`-ed this site. Shows how busy life have been lately. But it's great to be back and thanks for dropping by - 12K visits and 4 awesome followers! - not bad I believe.

Ok, I moved from a bit from pure UNIX admin/support work to a more interesting (niche) industry. Welcome to my world, welcome to Market Data! Very exciting! Before my current role, never heard of this (and a lot more), never crossed my mind it existed. How naive I am. Or we? So should I change the title of my blog now? Perhaps not. I still try (read: am too lazy) to get my hands dirty whenever I can.

Again, back to the new stuff I'm learning, specifically Reuters Market Data System or RMDS. As much as I want to discuss it, I still have to check to make sure I won't get sued (OK, I won't)! So for now, please bear with me as I compile some notes - hope it's useful for you, too!

P.S. You'll see NIX commands all over. Please leave a note or some sort if you can to correct or find better way to do it. All effort is appreciated.

A backgrounder (courtesy of Wiki):

Reuters Market Data System (RMDS) is an open market data platform provided by Reuters. RMDS is used to transport, integrate and manage financial data from stock exchanges and other data sources to end users (such as a bank or enterprise) using multicast or broadcast technology. The underlying Broadcast technology protocols are called Reuters Reliable Control Protocol (RRCP) and the Reuters Reliable Messaging Protocol (RRMP)

The latest version of RMDS is version 6, which uses Reuters Wire format as opposed to Marketfeed used in RMDS version 5.

OR check this site: http://thomsonreuters.com/products_services/financial/financial_products/a-z/enterprise_platform_for_real-time/

Rumor has it: something (ADH) will replace RMDS.

OK going back, basic structure of RMDS is:

Data Feed -> Source Distributor -> Sink Distributor -> Client App.

Perl-icious: Function that reads from a config file

2009-11-18T16:12:00.013+08:00

I was asked by my colleague to make a function he'll use for one of our scripts - to make our lives easier which I don't think won't happen unless THEY get rid of the night effin' shift! Anyway, it took me sometime to figure this out (I'm learning ok?!) and then I sought the help of a resident guru BE (his initial you idiot!).

Background:

I was to read the NetAgent configuration file and determine if the server is an MDH or P2PS (this is about RMDS). Another was from the RMDS config file to do the same, which means I got to make two.

Apologies but I cannot post config files here. But I believe you can easily understand how it process things.

Here is the code - as a standalone:

sub inNetAgentConfig {

my $host = shift;

my $conf_file = "/path/to/config.file";

open FH, $conf_file or die "Error: $!\n";

$/ = "ATTRIBUTES"; # set the line delimiter to ATTRIBUTES instead of default value of newline; see special variables

my @array = <FH>;

if ( grep {/$host/} $conf_file ) {

foreach my $line ( grep {/$host/} @array ) {

if ( $line =~ /GROUP=(\w+)/ ) {

return $1;

}
}
} else {
return "Non-existent";
}

}

Another function that I made:

sub inRmdsConfig {

my $host = shift;

my $rmds_file = "/path/to/config.file";

open FILE, $rmds_file or die "Error: $!\n";

while (<FILE>) {

chomp ( $_ );

if ( $_ =~ /^$host\*(\w+)\*serverId.?/ ) {

return $1;

} else {

return "Non-existent";

}

MySQL - an old friend: DELETE and OPTIMIZE

2009-11-14T17:04:00.008+08:00

Hey, how have you been? I know, it been sometime. Oops, I'm talking to myself again! A lot of things have happened and I could have posted some but not so excited. Not until now.

Background:

We are always receiving disk space alerts on a certain file system in a server owned by our team. What we usually do is to run a script (from crontab) which basically cleans up the MySQL log file. Yes, the nasty log files. But, I think, they forgot to consider having a script that trims the database/s - with entries that dates back 2006-07! Well, a colleague gave us a script that supposedly do this but it got some errors which, for now, is useless.

After checking with the senior engineers (read: oldies), it was agreed to delete some data and leave 30 days worth of records. As I dig deeper, I found out that one of the table have 177M rows - yes, that is M-illion! Hmm, it looked like I ate more than I could chew. But as usual, I know I am hard headed (sometimes, ok?) and I wouldn't give up just yet.

I used to do some maintenance when I was in my first job. I did some dumping, checking replications (my boss set it up, not me), deleting, optimize, etc. And it helped me - made me feel confident, speaking what experience can give you.

After some experiment - deletion, converting UNIX timestamp - I started my work. I deleted rows by the millions, about 10 in each run (I used ORDER BY and LIMIT functions here). You may wonder why not in one shot, well, the problem with DELETE is it locks the table and the table is being used by a monitoring tool which cannot be stopped for a long time. From SHOW PROCESSLIST, I can see a lot of INSERT statements queueing - and deleting 10M rows runs for 16+ minutes. I was so happy that I forgot that deleting doesn't equate to reducing disk space automatically. Just a side note, on the back of my mind, something tells me to mention it depends on how the table was created (?), not sure though. I'll read on this more I guess. Going back, I did reduce it to 55M+ and still see a 96% disk space usage. Outrageous! Doing some little research, I came up with a fragmented rows theory - well, originally not mine, of course, I read it somewhere. And to clean this mess up is to OPTIMIZE TABLE. But before doing it, I did some homework. What is the effect of optimization? I felt that something behind this good thing is an evil lurking! Again, the table is locked and it will create a .TMD file (temporary .MYD file - please read more on this) that can wreck havoc. It can fill-up the file system depending on how big it is and how much is left. So be very careful when doing optimization - I knew this first hand. I also ran CHECK TABLE [STATUS] which could give me some indicators if table needs some repair, or anything. 'Though, at times, this won't give you anything at all. From what I just did, it says everything is good. And yet, I got a ton of fragmented rows. Well, could be some limitation - again, read on.

After all these crappy steps, I was ready to corrupt the database. Oh before I forget, P-L-E-A-S-E if you have a test database or something, do these steps there and not on the prod. Trim the the database there then, a short downtime (stopping applications) for moving it back. So here are the simple steps I took:

1. I used this to get my target date as to set my limit later for deletion.

SELECT UNIX_TIMESTAMP('YYYY-MM-DD HH:mm:ss');

2. Now, we're ready to delete some rows with limits. This is IMPORTANT or the next thing you'll know, you just deleted the entire data!

DELETE FROM <tablename> WHERE timestamp < <number from SELECT statement here> LIMIT 10000000;

3. On this part, I optimize. You might wonder why now, not after the whole thing. Well, you run after everything depending on the amount of rows you just deleted. In this case, it just too darn big which could lock my table my a very long time (INSERTs queue up) and create a .TMD file too big that could overwhelm my file system which have a domino effect on other applications/processes that use it.

OPTIMIZE TABLE <tablename>;

Let this run to completion or it could render the table unusable or corrupted data! You've been warned. Of course, you can run this again to fix it or do a REPAIR TABLE. But who knows, you might also lose it all. As the S.A. says, "He who laughs last has a backup."

4. And then I am "maarte" so I add this. Its significance is here.

FLUSH TABLE <tablename>;

That pretty sums up what I just did. So long. And yes, please I'd like to hear from you. Corrections are always welcome. Cheers!

Scripting 101: Modification on the previous topic

2009-06-11T00:53:00.004+08:00

Before I modified the original script, I was advised by my colleague to take note of the date especially for the single digits, which was not addressed before. So, the corrections include re-assigning of variables, deletion of unwanted, throwing errors to null, etc.

Here they are:

1. CSV_DATE=`date '+%a %b %d'`

2. LIST_DATE was removed.

3. echo-es were deleted.

4. Errors were thrown to /dev/null; when added to cron, this will be re-directed to /dev/null as well.

5. mail was not working so I used EOF for the body:

/usr/ucb/mail -s "Subject here..." $EMAIL_ADD <<\

EOF

# Whatever the content between EOFs will be evaluated a regular formatted text (including blank lines and spaces) except for variable substitution such as here, which will output the result - the current date.

This will be a regular text.

$(date)

EOF

6. Since LIST_DATE was removed, I used 'find' to locate the most recent files that were modified, as:

for CSV in `find /path/of/files -name "some*.csv" -mtime 0\

2> /dev/null`

cp -p $CSV /some/httpd/location/

done

I hope, everything will be ok now after this week's test. It'll be submitted for cron-nization!

Scripting 101: Shell Script that uses Perl

2009-05-23T08:56:00.014+08:00

Good morning (very sleepy now!).. I promised to myself that I will modify the script that we're using to generate a report. I'm not familiar - yet - with the contents but the thing is, we run this manually every night and then copy it to another directory which make it available to the user/s. It's just another basic modification but I added some "twists" a safety pre-caution or stop unwanted operation - well, that's what I wish, at least.

The main script was made by "Someone Else" (There! I ain't saying it's mine.). In time, I'll interpret the Perl part here.

#! /usr/bin/sh

# This is a modified version.

# Author: Someone Else

# Modified by: ME

# Renamed: badname_PT.sh

# Date: 23 May, 2009

# Version: 1

CSV_DATE=`date | awk '{print $1,$2,$3}'`

E_NOTCD=43

E_OK=0

E_OTHER=45

EMAIL_ADD="name@domain.com"

FILE_DATE=`date '+%d_%m_%y'`

HOSTS="/path/to/host_list_PT"

LIST_DATE=`date | awk '{print $2,$3}'`

SCRIPTDIR="/var/tmp/jf"

export all

# On top of the original script, hour condition was placed to make sure that it runs only after 17:59 daily.

if [ `date +%H` -gt "17" ]

then

cd $SCRIPTDIR || {

/usr/bin/mailx -s "Can't change to $SCRIPTDIR; Please \

check permissions..." $EMAIL_ADD

exit $E_NOTCD

}

# This generates the reports; the heart of the script

for H_LIST in `cat $HOSTS`

echo "Extracting bad names from P2PS: $H_LIST"

# No more manual intervention in changing the dates (CSV_DATE was used - from current date)

rsh $H_LIST cat /path/to/some.log* | grep \

"$CSV_DATE" | grep ptrade | perl -n -e \

'if (m/^.* ([0-9]*:[0-9]*:[0-9]*).*Open Failure for ($.*:.*$) \

by ($.*$) at ($.*\/net$).*/) {$_= "$1,$2,$3,$4"; $_ =~ s/[\)|\(]//g;\

print "$_\n";}' > /var/tmp/PT_BadRequests.$H_LIST.csv

sleep 1

echo "Copying /var/tmp/PT_BadRequests.$H_LIST.csv /var/tmp/jf"

cp /var/tmp/PT_BadRequests.$H_LIST.csv /var/tmp/jf

echo "Copy completed for $H_LIST"

done

sleep 3

chmod 666 /var/tmp/jf/PT_*

rm /tmp/PT_BadNames_*

rm /some/httpd/html/PT_BadNames*

tar cvf /tmp/PT_BadNames_$FILE_DATE.tar ./PT*csv

compress /tmp/PT_BadNames*.tar

cp /tmp/PT_BadNames* /some/httpd/html

chmod 666 /some/httpd/html/PT_BadNames*

sleep 2

rm /tmp/PT_BadNames*

rm /var/tmp/jf/PT_BadReq*

# This was added to copy the newly generated CSVs from /var/tmp to /app/httpd/html site

if cd /var/tmp

then

for csv in `ls -l *csv | grep PT_BadRequests | grep "$LIST_DATE" \

| awk '{print $9}'`

cp -p $csv /some/httpd/html/Primetrade_BadRequests/

ls -l /some/httpd/html/Primetrade_BadRequests/$csv

done

# On completion, a mail will be sent to intended recipient/s.

/usr/bin/mailx -s "Bad Name PRIMETRADE Report is DONE" $EMAIL_ADD

exit $E_OK

else

/usr/bin/mailx -s "Can't change to /var/tmp; Please check \

permissions..." $EMAIL_ADD

exit $E_NOTCD

echo "Not yet..."

exit $E_OTHER

Also, if this script is run manually, which uses csh, copying each file would be "tedious". So, I made a FOR-loop to do it - very basic but syntax is tricky.

% set LIST_DATE=`date +%b" "%d`

% foreach csv (`ls -l /var/tmp/*csv | grep PT_BadRequests | grep "$LIST_DATE" | awk '{print $9}'`)

? do

? cp -p $csv /some/httpd/html/Primetrade_BadRequests/

? ls -l /some/httpd/html/Primetrade_BadRequests/$csv

? end

Parents' Wish

2009-05-09T14:25:00.003+08:00

A little tribute to our parents on Mother's Day.

To Our Dear Child:

On the day when you see us old, weak and weary ...
Have patience and try to understand us ...
If we get dirty when eating ... If we can't dress on our own ...
Please bear with us and remember the times we spent feeding you and dressing you up.
If, when we speak to you, we repeat the same thing over and over again ... do not interrupt us... listen to us.
When you were small, we had to read to you the same story a thousand and one times until you went to sleep.
When we do no want to have a shower, neither shame nor scold us ...
Remember when we had to chase you with your thousand excuses to get you to the shower?
When you see our ignorance of new technologies ... help us navigate our way through those worldwide webs.
We taught you how to do so many things ... to eat the right foods, to dress appropriately, to fight for your rights ...
When at some moments we lose the memory or thread of our conversation ... Let us have the necessary time to remember ... and if we cannot, do not become nervous ...
as the most important thing is not our conversatrion but surely to be with you and to have you listening to us ...
If we ever do not feel like eating, do not force us.
We know well when we need to and when not to eat.
When our tired legs give way and do not allow us to walk without a cane.
Lend us your hand, the same way we did when you tried your first faltering steps.
And when someday we say to you that we do not want to live any more, that we want to die.
Do not get angry. Some day you will understand. Try to understand that our age is not just lived but survived.
Some day you will realize that, despite our mistakes,
we always wanted the best for you and we tried to prepare the way for you.
You must not feel sad, angry nor ashamed for having us near you.
Instead, try to understand us and help us like we did when you were young.
Help us to walk... Help us to live the rest of our life with love and dignity.
We will pay you with a smile and by the immense love we have always had for you in our hearts.

We love you, child.
Mom and Dad

VirtualBox 101: Enable USB in Winhoes XP on Mac OS X host (same for Unices)

2009-04-26T08:59:00.011+08:00

Hello! Why do I let others keep on forcing me to do things that are against my will (yet I enjoy them)? Crap!

I don't want to jinx but my target for the year is a Solaris certification. I know, I know, I got a very long way to go and steep hill to climb but I'm not that easy to give up!

You might wonder, what the hell is the connection? Anyone care? Well, certainly no one but just want to shout a piece of my mind anyway - remember, I get to write whatever I want, ayt? This is to "force" me to really go for it for I will be judged from here on. Crap! Stressful!

Going back, I got some materials - Tats note, legally obtained - in preparation but the sad part is, it will run only in Winhoes (now, this is really crap!). I really got no choice but to install XP in my VirtualBox. I planned to share folders from my Mac but from what I read, it is still not supported (Note however that Shared Folders are only supported with Windows (2000 or newer), Linux and Solaris guests.). So, I'll use my flash drive (SanDisk Cruzer Micro - 4GB, 16GB looks awesome BUT it burns a deep hole in your pocket!). With this, I have to activate USB in the settings of my Guest OS. After I "checked" to enable it and run my guest OS, nothing showed up in XP. Neither it was detected in VirtualBox yet I can see it mounted in my Mac. I tried to create a Filter with empty details for I knew nothing about it, and still nothing. Now, I was forced to read the user guide (when everything doesn't work, check the manual.. perfect), and I found this to get the info I need for the Filter:

$ VBoxManage list usbhost

VirtualBox Command Line Management Interface Version 2.2.0

Host USB Devices:

UUID: 98ac0131-bc9e-48f1-8dd9-9512cba98fba

VendorId: 0x045e (045E)

ProductId: 0x0737 (0737)

Revision: 1.0 (0100)

Manufacturer: Microsoft

Product: Compact Optical Mouse 500

Address: p=0x0737;v=0x045e;s=0x000005f7ad04cd23;l=0x06200000

Current State: Unavailable

...

UUID: 796f5590-1ed2-45fc-9c98-6baec073ad0e

VendorId: 0x0781 (0781)

ProductId: 0x5406 (5406)

Revision: 2.0 (0200)

Manufacturer: SanDisk

Product: U3 Cruzer Micro

SerialNumber: 08777107D092855A

Address: p=0x5406;v=0x0781;s=0x0000022d92a681f1;l=0x26200000

Current State: Unavailable

Armed, I filled in the details and re-started XP and voila, it's now working. Couldn't be happier.

Scripting 101: Monitoring Autosys Jobs - Correction in SU_MAILER

2009-04-19T20:48:00.005+08:00

Whoa! My prayer was answered! Problem is solved!

Helpful shell scripting guru, from a forum I'm in, was able to address my dilemma for the duplicate entries. I created a confusion which unknowingly had a BIG impact in it's evaluation of regex. Without much-ado, here it is:

sed '/'"$JOBNAME"'/{x;/Y/!{s/^/Y/;h;d;};x;}' JOBLIST.txt

Perfect! I'm just waiting for the explanation. I'm lazy right now to search for these.

Scripting 101: Monitoring Autosys Jobs - Changes

2009-04-01T21:24:00.022+08:00

When I got to the office today, I started working on the planned changes for the script. 'Though not all, I was able to find a way to implement number 4 from my early list. So here it goes.

Let me establish first the fact that this is primarily designed to address the current setup of jobs that we're monitoring. We already know when it'll be executed so the list is pre-arranged according to time of execution. From which I have considered the changes that I did. And as always, this is a work in progress so bear with me.

First change that I did was to add a couple of lines in the JOB_CHECKER function after the SU_MAILER.

cp -p JOBLIST.txt TEMP.out

sed '1d' TEMP.out > JOBLIST.txt

It deletes job 1 from the JOBLIST.txt as not to waste time when doing a FOR-loop for jobs already marked as SU. It's kinda lame really but am looking for better way to cleanup this mess.

The second one is changing the condition in JOB_CHECKER function.

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]

was changed to [and DONE_JOB is removed]:

if grep -w $ONQUEUE_JOB EXCLUDEJOB.txt

Third is on WHILE-loop. The script exits when there's nothing left [to read] in the JOBLIST.txt. Hence, COUNTER is no longer needed.

while [ "$NO_JOBS" -gt "0" ]

...

It's still consuming a lot of CPU and I haven't figure this one out yet. To allow "others", after a FOR-loop, I put the script into sleep.

And lastly, a safety net was added in the SU_MAILER function. It'll erase duplicate entries from the exclusion list.

SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

awk '!x[$0]++' EXCLUDEJOB.txt > EXCLUDEJOB.txt

I hope I'm on track on improving my scripting skills; always looking forward for a new challenge.

********

This part was added on 15 April, 2009 at 21:33 SST.

The variable $NO_JOBS was removed. Instead, its value was placed in the WHILE-loop. It was late when I found out that it is NOT being updated. Also, there are times when some of the jobs listed may be invalid for one reason or another. With this, condition was added as well. So the code will be:

while [ `cat JOBLIST.txt | wc -l` -gt "0" ]

for JOBNAME in `cat JOBLIST.txt`

if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep "Invalid Job Name" 2>&1 > /dev/null

then

cp -p JOBLIST.txt VALJOB.out

sed -e "\|^$JOBNAME\$|d" VALJOB.out > JOBLIST.txt

continue

...

done

exit $E_DONE

SU_MAILER () is still in the works. I still can't find how to delete duplicate entries while leaving one of them. The "awk" part that previously used was scrapped. It's NOT up to the job. Of course, it's my fault. ;)

I really hope I'll have the flame to continue on this for I started on my new job and is being "forced" to take on Perl which seems great. Now I know that it ain't a scripting language but rather on the grey area between interpreted and compiled. Still a long wayyyyyy. See'ya round.

Scripting 101: Monitoring Autosys Jobs - Evolved Form

2009-04-01T09:23:00.023+08:00

Finally, here it is! I have this tested last night - on a PROD server [Linux only for some commands might NOT run on Solaris] - and it went well. After making some adjustments, it did behave as expected. Wow, I can't believe it. After a long while, I did it. Not of course without the valuable insights from my mentors - CJ and Mors - who gave their all [I hope!?]. I am a student and beginner in scripting. As for now, I'm still analyzing the logs to maximize performance for I have seen that it wastes so much time in loops. So anyone out there, please help me!

Here are some of what I've observed and future plans:
1. Run via cron; OR run it via "screen"
2. High CPU utilization
3. Integrate mutt in the script [not sure where but I'll try; it's a bit cool, ain't it?]
4. Instead of having an exclusion file, after a job is done, it will remove from the JOBLIST.txt and exit when nothing to read.

**********
#! /bin/bash

# Calling AUTOSYS environment
. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN

# Definition and initialization of VARIABLES.
E_NOTFOUND=66
E_MISSING=43
E_DONE=0
EMAIL_ADD=`whoami`
COUNTER=0
NO_JOBS=`cat JOBLIST.txt | wc -l`

# Initialization of EXCLUSION file; this will be used to exclude successful job

> EXCLUDEJOB.txt
/bin/touch EXCLUDEJOB.txt
if [ ! -w EXCLUDEJOB.txt ]
then
/bin/echo "EXCLUDEJOB.txt file does not exist and/or not writeable! Check write permissions on the directory." | /bin/mail -s "EXCLUDEJOB.txt Missing" $EMAIL_ADD
exit $E_NOTFOUND
fi

# This is a mailer function for successful jobs; once the job ended successfully, it will write to EXCLUDEJOB.txt before sending a mail
SU_MAILER() {
/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}' >> EXCLUDEJOB.txt

/bin/mail -s "FOS Job Monitor Update" $EMAIL_ADD >>EOF

****** Feed Status ******

Job ended SU-ccessfully!

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)

***** End of Status *****

EOF

} # End of SU_MAILER function

FA_MAILER() {

/bin/mail -s "FOS Job Monitor Alert" $EMAIL_ADD >> EOF

***** PLEASE check Job Status *****

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -d -J $JOBNAME)

$(/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -q -J $JOBNAME)

***** End of Status ****

EOF

} # End of FA_MAILER function

JOB_CHECKER() {
DONE_JOB=`grep $JOBNAME EXCLUDEJOB.txt`
ONQUEUE_JOB=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $1$2$3}'`

if [ "$DONE_JOB" = "$ONQUEUE_JOB" ]
then
continue
else
JOB_TIMESTAMP=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{sub (/:/,""); print $3}'`

if [ "$DATE_STAMP" = "$JOB_DATE" ] && [ "$TIME_STAMP" -ge "$JOB_TIMESTAMP" ]
then
if /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | grep -w SU 2>&1 > /dev/null
then

SU_MAILER
COUNTER=$[$COUNTER+1]

elif /sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | egrep -w "ST|AC|RU" 2>&1 > /dev/null
then
break
else

FA_MAILER
COUNTER=$[$COUNTER+1]

fi
fi
fi
} # End of JOB_CHECKER function

# This is basically the main function of the script; it will continue to run until all jobs listed in the JOBLIST.txt are successful
while [ "$COUNTER" -lt "$NO_JOBS" ]
do
for JOBNAME in `cat JOBLIST.txt`
do
JOB_TZ=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME -q | grep timezone | awk '{print $2}' | cut -d/ -f2`

if [ -z "$JOB_TZ" ]
then
/bin/echo "Timezone is NOT defined in $JOBNAME properties. Please check." | /bin/mail -s "Missing TIMEZONE" $EMAIL_ADD
continue
else

JOB_DATE=`/sbcimp/run/tp/CA/Autosys/v4.0/bin/autorep -J $JOBNAME | grep $JOBNAME | awk '{print $2}'`

case $JOB_TZ in
London)
DATE_STAMP=`TZ=GMT date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Zurich)
DATE_STAMP=`TZ=GMT-1 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-1 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Eastern)
DATE_STAMP=`TZ=GMT+5 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT+5 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Singapore|HongKong)
DATE_STAMP=`TZ=GMT-8 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-8 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Tokyo)
DATE_STAMP=`TZ=GMT-9 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-9 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

Sydney)
DATE_STAMP=`TZ=GMT-11 date +%m/%d/%Y`
TIME_STAMP=`TZ=GMT-11 date +%X | cut -d: -f1-2 | awk '{sub (/:/,""); print}'`;;

*)
/bin/echo "What else could go wrong? Notify the script owner... Stressful." | /bin/mail -s "FOlSe Alert. Something is missing..." $EMAIL_ADD
exit $E_MISSING;;
esac
fi

JOB_CHECKER

done # End of FOR-loop

# Sleep will be added here for about 5 minutes

sleep 300

done # End of WHILE-loop

exit $E_DONE

Solaris Cheat Sheet: Episode 1

2009-03-29T22:42:00.026+08:00

Yellow everyone! Guess who's back?!? Who?? No one..

It's been a very long time since that day [whatever it is]. I've been busy lately [honestly, I'm tired blogging] so why not drop by and post something that's already there. As always, these are for my reference. You can take whatever you could and encouraged to share some. By the way, I just finished reading Hajime No Ippo and looking forward for Takamura-san's fight. As you know, Ichiro-kun just won his OPBF title defense against Randy Boy Jr. [Philippines; and it's a lame name - sorry!]. But the question is, will Ippo vs. Ichiro really materialize? Can't wait either.

Ooops.. got carried away but after reading the above, I came across a good, known site that can really help me. So here it is the first installment of my Solaris cheat sheet! [My other script is on it's way - some refinement, always, and suggestions from my colleagues -.. if you're interested, keep an eye on it ;)]

Debugging
18. cat -v -t -e [file]
/* Show non-printing characters */
19. dumpadm -d swap
/* Configure swap device as dump device */
20. ld -l
/* Check if you have a particular library */
21. truss -f -p
/* Using multiple windows, this can be used to trace setuid/setgid programs */
22. truss executable
/* Trace doing of given command ( useful debugging ) */

Disk Commands
23. /bin/mount -F hsfs -o ro /dev/sr0 /cdrom
/* Mount an ISO 9660 CDROM */
24. /usr/bin/iostat -E
/* Command to display drives statistics */
25. du -ad /var | sort -nr
/* Report the the disk used in /var in reverse order */
26. du -k .
/* Report disk usage in Kilobytes */
27. du -sk * | sort -nr | head
/* Shows the top ten largest files/directories */
28. du -sk *|sort -k1,1n
/* Reports total disk space used in Kilobytes in present directory */
29. du -sk .
/* Report total disk usage in Kilobytes */
30. fdformat -d -U
/* Format diskette */
31. mount -F hsfs -o ro `lofiadm -a /export/temp/software.iso` /mnt
/* Mount an ISO Image */
32. newfs -Nv /dev/rdsk/c0t0d0s1
/* To view the superfblocks available */
33. prtvtoc /dev/dsk/c1t2d0s2 | fmthard -s - /dev/rdsk/c1t3d0s2

/* One-liner to copy a partition table */
34. prtvtoc /dev/rdsk/c0t0d0s2
/* Disk geometry and partitioning info */
35. prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2
/* Copy partition table from one disk to another */
36. quot -af
/* How much space is used by users in kilobytes */
37. volrmmount -i floppy
/* Mount a floppy or other media easily by its nickname. */

Driver Parameters
38. ndd /dev/ip ip_forwarding
/* Show the ip_forwarding variable in the kernel */
39. ndd /dev/ip ip_forwarding 1
/* Set the ip_forwarding variable in the kernel */
40. ndd /dev/ip \?
/* Show all IP variables set in the kernel */

File Manipulation
41. dos2unix | -ascii
/* Converts DOS file formats to Unix */
42. fold -w 180
/* To break lines to have maximum char */
43. split [-linecount] [file]
/* Split files into pieces */
44. [vi] : %s/existing/new/g
/* Search and Replace text in vi */
45. [vi] :set list
/* Show non-printing characters in vi */
46. [vi] :set nu
/* Set line numbers in vi */
47. [vi] :set ts=[num]
/* Set tab stops in vi */

File System
48. /sbin/uadmin x x
/* Syncs file systems and reboots systems fast */
49. awk ' END {print NR}' file_name
/* Display the Number of lines in a file */
50. cat /dev/null > filename
/* Zero's out the file without breaking pipe */
51. cksum [filename]
/* View the checksum value for the given file */
52. dd if=/dev/rdsk/... of=/dev/rdsk/... bs=4096
/* Make a mirror image of your boot disk */
53. df -k | grep dg| awk '{print $6}' |xargs -n 1 umount
/* Unmount all file systems in disk group dg */
54. fsck -F ufs -o b=97472 /dev/rdsk/c0t0d0s0
/* Check and repair a UFS filesystem on c0t0d0s0, using an alternate superblock */
55. fsck -F ufs -y /dev/rdsk/c0t0d0s0
/* Check a UFS filesystem on c0t0d0s0, repair any problems without prompting. */
56. fsck -F ufs /dev/rdsk/c0t0d0s0
/* Check a UFS filesystem on c0t0d0s0 */
57. gzip -d -c tarball.tgz | (cd /[dir];tar xf - ) &
/* Unpacking tarballs to diff location */
58. gzip -dc file1.tar.gz | tar xf -
/* Unpack .tar.gz files in place */
59. ln [-fhns]
/* Creating hard links and soft links */
60. ls -al | awk '$3 == "oracle" || $3 == "root" {print $9}'
/* List all file names by testing owner */
61. ls -l | sort +4n
/* List files by size */
62. ls -la | awk '{ print $5," ",$9 }' | sort -rn
/* File sizes of current directory */
63. ls -lR | awk '{total +=$5};END {print "Total size: " total/1024/1024 "MB" }'
/* Recursive directory size calculations in MB */
64. mkisofs -l -L -r -o [image-name].iso [directory]
/* Create an ISO image of a directory */
65. mount -F ufs -o rw,remount /
/* Used to remount root to make it writeable */
66. mount -o remount,logging /spare
/* Re-mount the ro file system rw and turn on ufs logging */
67. mount -f pcfs /dev/dsk/c0d0p1 /export/dos
/* DOS fdisk partition from Solaris */
68. mv [filename]{,.new_suffix}
/* Renaming file */
69. pax -rw . /newdir
/* Efficient alternative for copying directories */
70. prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2
/* Cloning Partitiontables */
71. rpm -q --queryformat '%{INSTALLPREFIX}\n' [packagename]
/* [Linux] Locate binaries */
72. tar cf - . | (cd /newdir ; tar xf -)
/* Recursively copy files and their permissions */
73. tar cvf filename.tar
/* Create a tape (tar) archive */
74. tar xvf filename.tar
/* Extract a tape (tar) archive */

75. X=$(wc -l < filename); echo $X

/* Count number of lines in a file into a variable (ksh) */

76. zcat < patch_file.tar.Z) | tar xvf -

/* Extracts the patch_file that is a compressed tar file */

77. zcat [cpio file] | cpio -itmv

/* Show the contents of a compressed cpio */

File Transfer
78. find . -depth | cpio -pdmv /path/tobe/copied/to
/* Fast alternative to cp -pr */
79. find . -follow | cpio -pdumL /path/tobe/copied/to
/* Copy with symbolic links to be followed */
80. get filename.suffix |"tar xf -"
/* Undocumented Feature of FTP */
81. ssh cd /some/directory \&\& tar cf - | ssh cd /some/direstory \&\& tar xvf -

/* Move any file(s) without actually touching them */
82. put "| tar cf - ." filename.tar
/* Undocumented Feature of FTP */
83. sendport
/* FTP command for transferring large numbers of files within the same control session */

General

84. /bin/printf '%d\n' '0x'
/* Converts hexadecimal number to decimal. */
85. /usr/bin/catman -w
/* Create windex databases for man page directories */
86. echo 'obase=16;255' | bc
/* Simple way to convert decimal to hex */
87. FQ_FILENAME=; echo ${FQ_FILENAME%/*}
/* Extract directory from fully-qualified file name. */
88. mailx -H -u
/* List out mail headers for specified user */
89. ps -ef | grep -i $@
/* Access common commands quicker */
90. set filec
/* Set file-completion for csh */
91. uuencode [filename] [filename] | mailx -s "Subject" [user to mail]
/* Send files as attachments */
92. xauth -f /home/${LOGNAME} extract - ${DISPLAY} | xauth merge -
/* Allow root to xdisplay after su */

Hardware
93. cfgadm
/* Verify reconfigurable hardware resources */
94. m64config -depth 8|24
/* Sets the screen depth of your M64 graphics accelerator */
95. m64config -prconf
/* Print M64 hardware configuration */
96. m64config -res 'video_mode'
/* Change the resolution of your M64 graphics accelerator */
97. prtpicl -v | grep sync-speed
/* Discover SCSI sync speed */

Kernel
98. /usr/sbin/modinfo
/* Display kernel module information */
99. /usr/sbin/modload
/* Load a kernel module */
100. /usr/sbin/modunload -i
/* Unload a kernel module */
101. /usr/sbin/sysdef
/* Show system kernal tunable details */
102. nm -x /dev/ksyms | grep OBJ | more
/* Tuneable kernel parameters */
103. update_drv -f [driver name]
/* Force a driver to reread it's .conf file without reloading the driver */

Memory
104. pagesize -a
/* Available page sizes for Solaris 9 */
105. prtconf | grep Mem
/* Display Memory Size of the local machine. */

Network Information
106. arp -a
/* Ethernet address arp table */
107. arp -d myhost
/* Delete an ethernet address arp table entry */
108. lsof -iTCP@10.20.2.9
/* Display open files for internet address */
109. named-xfer -z qantas.com.au -f /tmp/allip
/* Get All IP Addresses On A DNS Server */
110. ndd /dev/arp arp_cache_report
/* Prints ARP table in cache with IP and MAC address */
111. netstat -a | grep EST | wc -l
/* Displays number active established connections to the localhost */
112. netstat -a | more
/* Show the state of all the sockets on a machine */
113. netstat -i
/* Show the state of the interfaces used for TCP/IP traffice */
114. netstat -k hme0
/* Undocumented netstat command */
115. netstat -np
/* Similar to arp -a without name resolution */
116. netstat -r
/* Show the state of the network routing table for TCP/IP traffic */
117. netstat -rn
/* Displays routing information but bypasses hostname lookup. */
118. snoop -S -ta [machine]
/* Snoop for network packets and get size and time stamp entries. */
119. traceroute
/* Follow the route to the ipaddress */

Network Tuning
120. /sbin/ifconfig hme0:1 inet 10.210.xx.xxx netmask 255.255.0.0 broadcast 10.210.xxx.xxx
/* Virtual Interfaces */
121. /sbin/ifconfig hme0:1 up
/* Bring virtual interface up */
122. /usr/sbin/ndd -set /dev/hme adv_100fdx_cap 1
/* Nailling to 100Mbps */
123. ifconfig eth0 10.1.1.1 netmask 255.255.255.255
/* Add an Interface */
124. ifconfig eth0 mtu 1500
/* Change MTU of interface */
125. ndd -set /dev/ip ip_addrs_per_if 1-8192
/* To set more than 256 virtual ip addresses. */
126. ndd -set /dev/tcp tcp_recv_hiwat 65535
/* Increase TCP-receivebuffers on Sol2.5.1 systems with 100BaseTx */
127. ndd -set /dev/tcp tcp_xmit_hiwat 65535
/* Increase TCP-transmitbuffers on Sol2.5.1 systems with 100BaseTx */

Processes
128. /usr/proc/bin/ptree
/* Print the parent/child process 'tree' of a process */
129. /usr/proc/bin/pwdx
/* Print the working directory of a process */
130. /usr/ucb/ps -aux | more
/* Displays CPU % usage for each process in ascending order */
131. /usr/ucb/ps -auxww | grep
/* Gives the full listing of the process (long listing) */
132. coreadm -i core.%f.%p
/* Append program name and process id to core file names */
133. fuser -uc /var
/* Processes that are running from /var */
134. ipcs
/* Report inter-process communication facilities status */
135. kill -HUP `ps -ef | grep [p]roccess | awk '{print $2}'`
/* HUP any related process in one step */
136. lsof -i TCP:25
/* Mapping port with process */
137. pfiles
/* Shows processes' current open files */
138. pkill -n
/* Kill a process by name */
139. prstat -a
/* An alternative for top command */
140. ps -edf -o pcpu,pid,user,args
/* Nicely formatted 'ps' */
141. ps -ef | grep -i | awk '{ print $2 }'
/* Creates list of running PID by */
142. ps -ef | grep -i | awk '{ print $2 }'
/* Creates list of running PID by */
143. ps -ef | grep | grep -v grep | cut -c 10-15 | xargs kill -9
/* Find and kill all instances of a given process */
144. ps -ef | more
/* Show all processes running */
145. ps -ef|grep -v "0:00"|more
/* Gives you a list of any process with CPU time more than 0:00 */
146. ps -eo pid,args
/* List processes in simplified format */
147. ps -fu oracle|grep pmon
/* See which instances of Oracle are running */
148. top -b 1
/* Returns the process utilizing the most cpu and quits */

Resource Management
149. /usr/bin/ldd [filename]
/* List the dynamic dependencies of executable files */
150. /usr/proc/bin/pmap pid
/* Report address space map a process occupies */

Route Configuration
151. route add net 128.50.0.0 128.50.1.6 1
/* Add a route to the routing table */
152. route change 128.50.0.0 128.50.1.5
/* Changes the destination address for a route */
153. route delete net 128.50.0.0 128.50.1.6
/* Delete a route from the routing table */
154. route flush
/* Flush the routing table, which will remove all entries */
155. route get [hostname]
/* Which interface will be used to contact hostname */
156. route monitor
/* Monitor routing table lookup misses and changes */

Searching Items
157. cat | awk '{if (substr($1,1,1) == '#') print $0 }'
/* Print all lines in a file beginning with a specific character */
158. egrep "patterna|patternb"
/* Search for multiple patterns within the same file */
159. find -name "" -exec rm -rf {} \;
/* Recursively finds files by name and automatically removes them */
160. find . -type f -print | xargs grep -i [PATTERN]
/* Recursive grep on files */
161. find . ! -mtime - | /usr/bin/xargs rm -rf
/* Finds and removes files older than specified */
162. find . -exec egrep -li "str" {} \;
/* Find a string in files starting cwd */
163. find . -mtime -1 -type f
/* Find recently modified files */
164. find . -type f -exec grep "

" {} \; -print
/* Find files (and content) containing

within directory tree */
165. find . -type f -exec grep -l "

" {} \;
/* Find files (and content) containing

within directory tree */
166. find ./ \! -type f -exec ls -l {} \;|grep -v '^[l|p|s|-]'|grep -v 'total' | wc -l
/* Find number of directories under the current directory */
167. find / -fstype nfs -prune -o fstype autofs -prune -o -name filename -print
/* Find without traversing NFS mounted file systems */
168. find / -mtime <# of days>
/* Find files modified during the past # of days */
169. find / -perm -2 -a ! -type l
/* Find files writable by 'others' */
170. find / -type f |xargs ls -s | sort -rn |more
/* List files taking up the most system space */
171. find / -user
/* Find all files owned by */
172. find / -xdev -type f -ls|sort -nr -k 7,7
/* Find largest files in a file system */
173. find / | grep [file mask]
/* Fast way to search for files */
174. find /proc/*/fd -links 0 -type f -size +2000 -ls
/* Find large files moved or deleted and held open by a process */
175. grep /var/sadm/install/contents| awk '{ print $1 ' ' $10 }'
/* Find which package contains a particular file */
176. ls -lR | grep
/* Fast alternative to find. */
177. pkgchk -l -p /absolute/path/todir
/* Which package does this file belong to? */

Security
178. crypt abc && rm abc.cr
/* Decrypting a file that has been encrypted */
179. crypt abc.cr && rm abc
/* File encryption with crypt */
180. echo 'Please go away' > /etc/nologin
/* Stops users logging in */
181. find / -perm -0777 -type d -ls
/* Find all your writable directories */
182. find / -type f -perm -2000 -print
/* Find all SGID files */
183. find / -type f -perm -4000 -print
/* find all SUID files */
184. getpwenc [encryption scheme] password
/* Genrate passwords for LDAP Using 'getpwenc' Utility */
185. trap 'exit 0' 1 2 3 9 15
/* Trap specific signals and exit */
186. vi -x [filename]
/* Encrypt a file with vi editor */

Setting Term Options
187. stty erase ^?
/* Set the delete key to delete a character */
188. stty erase ^H
/* Set the backspace to delete a character */
189. stty sane
/* Reset terminal after viewing a binary file. */
190. tput rmacs
/* Reset to standard char set */

Snoop
191. snoop -d pcelx0
/* Watch all the packets on a device */
192. snoop -i /tmp/mylog -o /tmp/newlog host1
/* Filter out all the host1 packets and write them to a new logfile */
193. snoop -i /tmp/mylog -v -p101
/* Show verbose info on packet number 101 in the logfile */
194. snoop -i /tmp/mylog host1 host2
/* View packets from a logfile between hosts1 and host2 */
195. snoop -o /tmp/mylog pcelx0
/* Save all the packets from a device to a logfile */
196. snoop -s 120
/* Return the first 120 bytes in the packet header */
197. snoop -v arp
/* Capture arp broadcasts on your network */
198. snoop port [port-number]
/* Monitor particular port for traffic */

Swap Files
199. mkfile -nv 10m /export/disk1/myswap
/* Makes an empty 10 Megabyte swapfile in /export/disk */
200. mkfile -v 10m /export/disk1/myswap
/* Makes a 10 Megabyte swapfile in /export/disk */

Swap Space
201. swap -a /export/disk1/swapfile
/* Add a swap file */
202. swap -d /dev/dsk/c0t0d0s4
/* Delete a swap device */
203. swap -l
/* List the current swap devices */
204. swap -s
/* List the amount of swap space available */

System Configuration
205. /usr/sbin/eeprom auto-boot? false
/* Changes eeprom autoboot? setting without going to Ok prompt */
206. /usr/sbin/eeprom diag-switch? true
/* Set the system to perform diagnostics on the next reboot. */
207. /usr/sbin/eeprom local-mac-address?=true
/* Multiple Port Network Card Setting */
208. /usr/sbin/grpck
/* Check /etc/group file syntax */
209. /usr/sbin/pwck
/* Check /etc/passwd file syntax */
210. /usr/sbin/sys-unconfig
/* Clear host specific network configuration information */
211. /usr/sbin/useradd
/* Add a new user to the system */
212. drvconfig ; disks
/* Adding hot-plug disks to system */

System Information/Monitoring
213. /bin/echo "0t${stamp}>Y\n<Y=Y" | adb

/* Convert UNIX timestamp to something human-readable */
214. /usr/sbin/eeprom
/* Show eeprom parameters */
215. /usr/sbin/prtconf -vp
/* Show system configuration details */
216. coreadm -e log
/* Report global core */
217. grep "\-root" /var/adm/sulog | grep -v \+ | tail -25
/* List most recent attempts to switch to superuser account. */
218. isainfo -bv
/* Quickly checkout if machine is in 32 or 64 bit mode */
219. last
/* Tells who was or still is on the system */
220. logger -i
/* Log the process ID */
221. prtconf -pv | grep banner-name |awk -F\' ' { print $2 } ' | head -1
/* Show actual model name of machine */
222. prtdiag -v
/* System Diagnostics */
223. prtpicl -v | grep wwn
/* A command to find persistent binding of storage */
224. psradm -f [processor id]
/* Take processor offline */
225. psrinfo | wc -l
/* Display number of processors */
226. sar -u
/* Report CPU Utilization */
227. sar [ -aA ] [ -o filename ] t [ n ]
/* Provides cumulative reports about system activity. */

228. telnet 13 | grep ':'

/* Get the time on remote Unix machine */
229. uname -a
/* Displays system information */
230. uname -X
/* Displays system information */
231. vmstat 10
/* Displays summary of what the system is doing every 10 seconds */
232. who -b
/* Displays the date of the last system reboot. */
233. ypcat hosts | sort -n -t. +0 -1 +1 -2 +2 -3 +3 -4
/* Take the input of "ypcat hosts" or "cat /etc/inet/hosts" and sort by IP. */

Veritas NetBackup Tutorial: Changes in bp.conf

2009-02-04T01:38:00.012+08:00

Aha! Like a li'l child on his first day in school, I was excited. That's me. When I encounter problems/errors I can't help but get excited looking for solutions, the flame of eagerness to learn. Of course, these are limited instances. As for today, it will on modifications in bp.conf. A good friend asked me if changes be made on the config file needs to re-read them. Since I am NOT familiar with this nor had the experience, I provided no answer [sorry but, my shift is up and just want to go home, ha ha]. But, this doesn't mean that I'll just lay down and die. Not me! When I got home, I searched for the solution. I even asked taqadus from his blog for some advise [Thank you!]. And my conviction is final when I got into forums and been reading suggestions/solutions the same over and over again. Plus documentation from Symantec and part of which reads:

...
Note: This command will only detect and incorporate some bp.conf configuration changes. Many NetBackup configuration changes, such as adding or changing a SERVER entry, require a complete stop and start of the NetBackup daemons. If the bp.conf configuration change does not take effect, then the NetBackup daemons must be restarted. To do so, when no backups or restores are running, execute the commands:

# /usr/openv/netbackup/bin/goodies/netbackup stop
# /usr/openv/netbackup/bin/goodies/netbackup start

Scripting 101: Purging Files

2009-01-29T00:45:00.013+08:00

Ei-lo! 'Zup? I'm beginning to enjoy this I guess: scripting. Unexpected, I got a ping from a former colleague, asking from a noobs [read: me] regarding scripting. Without blinking, I accepted the challenge. I was asked to check on the script she made for purging files. After checking on the requirements, given, and asking for advise from a scripting guru [Morsgang Freeman], below are the revised forms. Btw, the first one I made really sucks [I hope these won't :(]. As of writing, these are not tested, not 'til tomorrow when she receives the email/script.

1. First is to create a file, say SUBDIR.txt with the ff. contents:

MERGE/REJECTS/ARCHIVE
MERGE/REJECTS/ARCHIVERAW
MERGE/FILTERED/ARCHIVE
MERGE/FILTERED/ARCHIVERAW
INSERT/REJECTS/INSERTED
INSERT/FILTERED/INSERTED
FTP/FTP_ARCHIVE

2. I made two variations of the script:
a. Version 1

#!/bin/bash

export LOGDIR=/ramsys_data/bmcps/ramsys/
export DIRLIST="ALTL AMA5 CAMA FREE NRTL SEGR UTSS"

for XXX in DIRLIST
do
cd $LOGDIR
if [ "$PWD" = "$LOGDIR" ]
then
for PURDIR in `cat SUBDIR.txt`
do
# A log file can be added here [good practice] to list the files to be deleted...
find $XXX/$PURDIR -type f -name "*" -mtime +5 -exec rm {} \;
done
else
echo "Check $USER\'s permissions; cannot change to $CURDIR."
# You can add a mailer here for notification...
exit 1
fi
done

b. Version 2

#!/bin/bash

LOGDIR=/ramsys_data/bmcps/ramsys/
DIRLIST="ALTL AMA5 CAMA FREE NRTL SEGR UTSS"

cd $LOGDIR || {
echo "Check the permissions; can't change to $CURDIR."
# You can add a mailer here for notification...
exit 1
}

for XXX in DIRLIST
do
for PURDIR in `cat SUBDIR.txt`
do
# A log file can be added here [good practice] to list the files to be deleted...
find $XXX/$PURDIR -type f -name "*" -mtime +5 -exec rm {} \;
done
done

If this will be run in cron, e.g. everyday at 0100H, we can make it as:

0 1 * * * ( scriptname_here.sh > /if_wanted/can/store/to/logfile 2>&1 ) &

Or throw it to /dev/null. Whatever suits you.

Scripting 101: Monitor Long-running Autosys Jobs

2009-01-28T00:00:00.009+08:00

HELLo World! It's been sometime since I made a "working" script. The last time I remember I created is about 2 years ago [I guess]. I lost interest in programming for a while for some work-related reason and just had this renewed interest a few weeks back. Without much a-doo, here it is: [Sorry 'bout the format, I'm not in the mood to edit it - for now]

#!/bin/bash
### Calls some environment settings to use Autosys
. /sbcimp/shared/config/CA/Autosys/v4.0/prod/autouser/shellPLN

### Declaration of VARIABLES
declare -i SKIPTIME
declare -i NOMIN
declare -i NOSEC
ERRLOG=/home/$USER/Err.log
E_NOTFOUND=66

### Prompts the user for INPUTS
read -p "Enter the Job Name: " JOBNAME
read -p "Enter the time interval between display (in seconds): " SKIPTIME

### Check to make sure "interval" is integer; don't know what else to do...
if [ $SKIPTIME -le "0" ]
then
echo "Make sure time interval is greater than zero or a number."
exit 1
fi

### This part acts as catch for INVALID Jobs
CHKINVJOB=`autorep -J $JOBNAME`
echo $CHKINVJOB > $ERRLOG

if [ ! -f $ERRLOG ]
then
echo "Error log file does not exist! Check write permissions on the directory."
exit $E_NOTFOUND
exit 1
fi

more $ERRLOG | grep Invalid
if [ $? == "0" ]
then
more $ERRLOG
echo "--- Clearing $ERRLOG ... ---"
cat /dev/null > $ERRLOG
exit 1
fi

### This part checks for the status of the Job
while true
do
autorep -J $JOBNAME | sed -n '5p' | grep -w SU > /dev/null
if [ $? == "0" ]
then
clear
echo "---- Feed Status ----"
echo
echo "***** Job ended SU-ccessfully! *****"
autorep -J $JOBNAME
echo "---- End of Status ----"
exit 0
elif [ $? == "1" ]
then
autorep -J $JOBNAME | sed -n '5p' | egrep -w "ST|AC|RU" > /dev/null
if [ $? == "0" ]
then
if [ "$SKIPTIME" -gt "60" ]
then
NOMIN=`expr $SKIPTIME / 60`
NOSEC=`expr $SKIPTIME % 60`
clear
echo "---- Feed Status ----"
echo "It is still processing... will report in approx. $NOMIN minutes and $NOSEC seconds."
sleep $SKIPTIME
autorep -J $JOBNAME | sed -n '5p'
echo "---- End of Status ----"
else
clear
echo "---- Feed Status ----"
echo "It is still processing... will report in approx. $SKIPTIME seconds."
sleep $SKIPTIME
autorep -J $JOBNAME | sed -n '5p'
echo "---- End of Status ----"
fi
else
clear
echo "---- Feed Status ----"
echo "Check for any ERROR."
autorep -J $JOBNAME
echo "---- End of Status ----"
exit 1
fi
fi
done

exit 0

Mirroring a Root Volume or Disk, and /boot

2009-01-04T22:09:00.026+08:00

I slept late last night, or may be late this early morning. Does it make sense? Anyway, when I woke up, I prepared my food. Forget it. Alright, here we go.

I was doing my morning ritual - if got no work - checking my mails, visiting PACLAND, NBA, going thru today's headlines including stock market? Freakin' sick, ayt?! Can't help it esp. in times like these and working in a bank!

Whew! I find it funny writing about this. Another problem is, I was worrying more on the title than the content.

Seriously, here it is.

I was reading RHEL 5 Deployment Guide. Actually, I was continuing. I was considering of skipping the LVM part but decided to go for it anyway. And something, which triggered this blog, caught my attention: The /boot/ partition cannot reside on an LVM volume because the GRUB boot loader cannot read it.

Interesting point. I admit, I was more of an HP-UX LVM guy. Shame on me for the confusion. I begun to ask a former colleague - Che, kudos - to execute some commands for us to verify if it the same with HP-UX. I was so pissed at myself and at the same time wanted to laugh that I was not sure which command to say. Finally, I got me a `setboot`:

[root@server1032:/root]
# setboot
Primary bootpath : 0/1/1/0.0.0
Alternate bootpath : 0/1/1/0.1.0

Autoboot is ON (enabled)
Autosearch is ON (enabled)

Which at that point, doesn't answer the question nor mean anything [yet]?!. So, I Googled it with key words: HP-UX LVM /boot. See? /boot in HP-UX? Another evidence that I wasn't thinking right - since this is an HP-UX issue, I should have gone to ITRC.

Too, late.

... and there it is [as I end up with ITRC]. How can I forget it? It's included in vg00. In fact, there are commands such as mkboot and -B for pvcreate. And please take note of lvlnboot, there is this line that says: Boot.

On the side note, for a bad boot, mirrored disk replacement case, here's how it is done [PA-RISC and Integrity] - this is available on HP's docs:

---
Mirroring the Root Volume on PA-RISC Servers

To set up a mirrored root configuration, you must add a disk to the root volume group, mirror all the root logical volumes onto it, and make it bootable. For this example, the disk is at path 2/0/7.15.0 and has device special files named /dev/rdsk/c2t15d0 and /dev/dsk/c2t15d0.

1. Use the insf command with the -e option to make sure the device files are in place. For example:

# insf -e -H 2/0/7.15.0

You should now have the following device files for this disk:
/dev/dsk/c2t15d0 The entire disk (block access)
/dev/rdsk/c2t15d0 The entire disk (character access)

2. Create a physical volume using pvcreate with the -B option:

# pvcreate -B /dev/rdsk/c2t15d0

3. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t15d0

4. Use the mkboot command to set up the boot area:

# mkboot /dev/rdsk/c2t15d0

5. Use the mkboot command to add an autoboot file to the disk boot area. If you expect to boot from this disk only when you lose quorum, you can use the alternate string “hpux –lq” to disable quorum checking:

# mkboot –a “hpux” /dev/rdsk/c2t15d0

1. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t6d0 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00038 current /dev/vg00/lvol2 00000
00550 current /dev/vg00/lvol3 00000
00583 current /dev/vg00/lvol4 00000
00608 current /dev/vg00/lvol5 00000
00611 current /dev/vg00/lvol6 00000
00923 current /dev/vg00/lvol7 00000
01252 current /dev/vg00/lvol8 00000

In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t15d0

6. Update the root volume group information:

# lvlnboot -R /dev/vg00

7. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

8. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 2/0/7.15.0

9. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t15d0

where l denotes LVM.
---

Mirroring the Root Volume on Integrity Servers

The procedure to mirror the root disk on Integrity servers is similar to the procedure for PA-RISC servers. The difference is that Integrity server boot disks are partitioned; you must set up the partitions, copy utilities to the EFI partition, and use the HP-UX partition device files for LVM commands.

For this example, the disk is at hardware path 0/1/1/0.1.0, with a device special file named /dev/rdsk/c2t1d0.

1. Partition the disk using the idisk command and a partition description file.

a. Create a partition description file. For example:

# vi /tmp/pdf

In this example the partition description file contains:
3
EFI 500MB
HPUX 100%
HPSP 400MB

b. Partition the disk using idisk and the partition description file created in step 1a:

# idisk -f /tmp/pdf -w /dev/rdsk/c2t1d0

To verify that your partitions are correctly laid out, run the following command:

# idisk /dev/rdsk/c2t1d0

2. Use the insf command with the -e option to create the device files for all the partitions. For example:

# insf -e -H 0/1/1/0.1.0

You should now have the following device files for this disk:
/dev/dsk/c2t1d0 The entire disk (block access)
/dev/rdsk/c2t1d0 The entire disk (character access)
/dev/dsk/c2t1d0s1 The EFI partition (block access)
/dev/rdsk/c2t1d0s1 The EFI partition (character access)
/dev/dsk/c2t1d0s2 The HP-UX partition (block access)
/dev/rdsk/c2t1d0s2 The HP-UX partition (character access)
/dev/dsk/c2t1d0s3 The Service partition (block access)
/dev/rdsk/c2t1d0s3 The Service partition (character access)

3. Create a physical volume using pvcreate with the -B option. Be sure to use the device file denoting the HP-UX partition.

# pvcreate -B /dev/rdsk/c2t1d0s2

4. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t1d0s2

5. Use the mkboot command to set up the boot area. Specify the -e and -l options to copy EFI utilities to the EFI partition, and use the device special file for the entire disk:

# mkboot –e –l /dev/rdsk/c2t1d0

6. Update the autoboot file in the EFI partition.

a. Create an AUTO file in the current directory. If you expect to boot from this disk only when you have lost quorum, you can use the alternate string “boot vmunix –lq” to disable quorum checking:

# echo “boot vmunix” > ./AUTO

b. Copy the file from the current directory into the new disk EFI partition. Make sure to use the device file with the s1 suffix:

# efi_cp -d /dev/rdsk/c2t1d0s1 ./AUTO /efi/hpux/auto

7. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t0d0s2 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00010 current /dev/vg00/lvol2 00000
00138 current /dev/vg00/lvol3 00000
00151 current /dev/vg00/lvol4 00000
00158 current /dev/vg00/lvol5 00000
00159 current /dev/vg00/lvol6 00000
00271 current /dev/vg00/lvol7 00000
00408 current /dev/vg00/lvol8 00000

In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t1d0s2

8. Update the root volume group information:

# lvlnboot -R /dev/vg00

9. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

10. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 0/1/1/0.1.0

11. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t1d0s2

where l denotes LVM.
---

My first question has been answered. But, as a former S.A. and UNIX support engineer [for a huge infra], a redundant connection and/or backup is always in the back of our minds. As they say, "He who laughs last, has a backup."

So, the second question is, if /boot is not under the control of LVM, what is a good way to have a redundant connection? I don't have enough experience with this so, I turn to an old friend who has or had. We talked of RAID: software and hardware. For this case, he suggested RAIDS: 1, 5, or 1+0. In case you're server have a RAID controller, it is recommended to go for hardware. This and backup, he said, is enough to get you through.

Basically, 100MB is enough for /boot so, the rest of the disk, in case you decide to use LVM, can be brought under its control. While some others separate / as well for faster recovery process, he continued, for there were cases where the root VG, that contains the default file systems, is not detected. And this is a lengthy process of recovering it. It's a big no-no in critical systems, where downtime is associated with business loss [read: money].

So, I guess, that's it. I'm quite satisfied with the answers, and then some.

UPDATED: Veritas NetBackup Tutorial: some troubleshooting commands, etc.

2008-12-29T15:43:00.074+08:00

Supposedly, I'll let several days to pass by before making this follow up but I was thinking, it could turn into weeks or even months before I can find time. So, before that happens, I'm going to publish the sequel this early. I don't know but, at times, no, often, I easily forget the details about things BUT it won't take time before I remember it anyway. Just a bit of focus and re-channeling of some energies and time and am back on track again. Ok, 'nuff with the gibberish.. moving forward...

Very early today, I laid down the basic config and executable files needed by NBU to run, at least in a UNIX environment. What's next are some basic troubleshooting commands that you might find useful in finding errors [code] generated by failure, or determine the culprit behind it.

And before we proceed any further, you may find it missing, i.e. topic about installation. As of this writing, I don't have a background in NBU installation. My experience is mostly on the maintenance part. So, please spare me of such questions. But just like any eager student, I'm always willing to learn and craves for knowledge.. at times!

Back!

I'll try to group these according to scenarios when they're needed but no guarantees.

# bpadm = text-based user interface; much quicker than jnbSA [Java-based]; can be used for restoration

# vmoprcmd -d = list of tape drives that are busy with active jobs; status [TLD, TLD-DOWN, AVR, PEND-TLD]; tapes mounted on to drive

# vmoprcmd -hoststatus -h hostname = should get similar o/p as below

Host `hostname` is ACTIVE

# vmoprcmd -h hostname -dps = will report if drive is SHARED; status should be UP

# vmdareq -driveinfo = check which drive is RESERVED or AVAILABLE

# tpconfig -d = tape device files configured for the machine and its status; helpful in identifying EMM

# robtest = to run SCSI pass-through commands; scans drive within library to find empty/full slot IDs; unloading and/or moving tapes across

# bptestbpcd -host hostname [-client clientname] [ -verbose | -debug] = check communication between client and server; to find communication problem; can be executed either from Master or Client

# vmquery -m [-ev in version 3.4] mediaID = tape density, slot number, volume pool assigned, tape location [on/offsite], vault session ID for tape location if offsite; please take note of the result robot type, if it is set to TLD-Tape Library, the tape is in silo, otherwise, it'll show NONE- Not Robotic which is the opposite

# bpgetmedia -p Scratch [| wc -l]

EH0466 8 800 20

EH0901 8 800 20

...

# echo "s d" | tldtest -r /dev/rac/c10t6d0 | grep mediaID = checks for media availability in the robot; -r here refers to the device file/path of the robot [at least for HP-UX]; this shows the drive ["s d"] or ["s s"] for slot

# tpclean -C drivenumber = clean a drive

# vmcheckxxx -rt tld -rn robotnumber = lists the tapes [TLD] currently in the robot; will use current host if neither -h hostname nor -rh robothost is specified

# /usr/openv/netbackup/bin/admincmd/bpdbjobs -report | grep clientname | grep policyname | grep schedule = generates a report of jobs that are done - successful or not, queued, and/or active

# bperror -jobid jobnumber_from_bpdbjobs_o/p -U = provides details about a particular job that ran; usually includes the files/directories being backed up, media used, and error - if failed

# bperror -L -backstat -columns -hours_ago HH | awk '{if (/CLIENT/) ORS="\n" ; else ORS=" "} {print $0}' | grep -i clientname | awk '{print $1, $2, $4, $5, $6, $12, $14, $16, $17, $18, $19}' | grep -i policyname = this is an alternative way of getting the error code from logs

# bpdbjobs [-restart | -cancel | -cancel_all] jobID_number

# telnet clientname bpcd = to check if bpcd daemon is accepting connection; usually executed from Master

# netstat -an | grep -i bpcd = should return bpcd with LISTEN status; see previous blog [and click the Google ads as well, he he]

# bpclntcmd [-ip IP_of_Master | -hn hostname_of_Master] -pn = hostname resolution; resolve multiple host interfaces on clients

# bpplclients | grep -i clientname = lists the client - if present - that is assigned to the Master

# bppllist | grep -i policy_pattern = shows the list of policies accdg. to pattern

# bppllist -byclient clientname -U = lists the policy defined or being used by a particular client

# bppllist policy_name -L -U = a detailed display of properties of a particular policy

# bpdbm -ctime 1161224987 = this will convert it to common format of date and time

1161224987 = Thu Oct 19 10:29:47 2006

# bpimagelist -client clientname -backupid clientname_numberFromANF = I'm not sure if ANF setup is true for all

# bpimagelist -client clientname -d MM/DD/YYYY HH:mm -e MM/DD/YYYY HH:mm -U [-m mediaID] [-L] = this is executed from the Master; checks for saved images from the date specified; note that images saved here are may be limited [not sure 'though depending on the infra setup]

# bpmedialist -U mcontents -m mediaID = list media contents

# bpmedia [-freeze | -unfreeze | -suspend] -ev mediaID

# bpexpdate -d 0 -ev mediaID -force = expire a tape

And lastly, you can run a manual backup as:

# echo "/var/opt" > /home/user01/nbfile

# bpbackup -i -c classname -s schedule -L /path/to/log/file -f /home/user01/nbfile

# bpbackup -p policyname -s schedule -L /path/to/log/file /usr/openv/netbackup/bp*

You might be interested on the what tape device files means such as this:

/dev/rmt/cXtYdZBESTnb

where:

BEST = operational capabilities required including the highest density/format and data compression, if supported

n = no rewind

b = Berkeley style; after file closes, tape is not repositioned in anyway

Please note that this was added on 31 December, 2008 @ 17:53:35...

I almost forgot regarding ejecting a tape, which is as crucial as any other task. I'm not sure but I knew I read it somewhere: eject the tape on the robot first before doing it via `nbmenu`. Anyway - sorry, I accept the fact that I do forget things, often -, here is the way to do it:

1. Access the robot via `robtest` [careful with this command, for it can do nasty things to your robot].
2. Before doing anything stupid, please consider the tape if NOT used by other backup jobs.
3. Select from the classification of tapes you want to move [TLD 0, etc.].
4. Execute the following:

"s d" - to check on the contents of the tape drives or identify the drives that has tape (Contains Cartridge =yes, Barcode = XXXXX)
"s s" - check on the content of the library or identify the empty slots (re-inventory)
"m d# s#" - move the tape from drive to slot
"s d" - to verify that the drive is empty after move
"s s" - to verify that the slot has the tape

5. Quit.
6. Now, you can go to the NBU level - which is pretty straightforward - to eject the tape via textual user interface [nbmenu] or CLI:

vmchange -h Master -multi_eject -res -ml list_of_tapes_delimited_by_colon -rt robot_type -rn robot_number -rh MM -sc -verbose

So I guess, this is it. Hope it will help. If you find anything wrong with these tutorials or guides [means, included are the previous], please do leave a message, and I'd be more than happy to check and learn from it. Parking..

Veritas NetBackup Tutorial: config, executable files

2008-12-29T00:32:00.021+08:00

Previously on Netbackup [cool; sounds like Heroes]... we discussed the overview on how NBU works. Going forward, we'll do check out the directories and files needed and/or configured for it to work.

Well, the one of the most - IF not the most - important files is bp.conf located in /usr/openv/netbackup. This file [which can be found accross setup] tells how a machine is configured: Master, Media Manager, and/or Client. Please also note that a machine can be configured as follows:

Master-MM-Client-in-one
MM-Client-in-one
Client

And here is a sample of a basic entry:

$ more /usr/open/netbackup/bp.conf
SERVER=[name of Master]
SERVER=[name of Media Manager]
...[list of other servers]
EMMSERVER=[name of EMM; mostly Master - depending on setup]
CLIENT_NAME=[name of Client; depending on the role as defined above]
... [additional options follows]

Another config file you might want to check-up is /usr/openv/volmgr/vm.conf which contains:

MM_SERVER_NAME=[name of MM]

In addition to these files, we are also considering native UNIX files and ensure that they're properly configured to allow NBU to run:

/etc/services - defines service names and corresponding ports
/etc/inetd.conf
/dev - device files directory which will identify the robots and tapes

Forgive me, I've been busy with specifics and gravely forgot the parent of it all. Almost all of the files: config and commands are located in /usr/openv for Unices or [install directory]\VERITAS for Windows.

So what we're talking here are:

../netbackup - contains NBU, VolMgr binaries, NBU DB
../db/data - EMM & NB databases
../netbackup/db - NBU DB of class, schedules, images, etc.
../netbackup/logs - log files

Here are the daemons, in addition to what was listed before:

tldd - started with ltid robotic daemon one on each MM server
tldcd - started with ltid; talks to the robot
bpbrm - backup and restore manager
bptm - tape manager
bpdm - disk manager
avrd - bar code reader

Next, we'll go to basic troubleshooting of common issues encountered.

Veritas NetBackup Tutorial: overview

2008-11-17T08:24:00.026+08:00

It's been a very long time.. always is. The hard part for this kind of blog is you only get to write when something interesting pops up or an error was encountered [and you get to solve it - right] or a new experience.

Well, I never really had the time to internalize this software when I was working with it some few months ago, in my previous job. Just last week [not true anymore; the time here is relative to when I was preparing for a interview - published December 28, 2008 @ 16:19.xx], our paths has crossed, again.

The sudden interest rose from a technical panel [or panel technical?] interview that I've been through. Well, I was satisfied with the result for at least I was able to address all the issues that they've thrown at me, 3 of them. You can't help it but even though I was advised that it has something to do with Veritas NetBackup [NBU], it always will cross with UNIX or Linux. Surprisingly, including myself, I have at least a deeper basic understanding of how NBU works than ever before [and I hope that this is right].

Please note that this so-called tutorial is more aimed to those who have worked with or working with NetBackup yet do not understand how it works. And another thing is, this is on how I understand how it works, which is not necessarily true. But so far, no one questions it, so it must be true. Before it'll cause you giddiness, let's proceed...

****** START ******
In order for a successful backup to run, in NBU environment, we need to complete The Triangle - Master, Media Manager, and Client - well on its simple state. 'Though we also have another entity such as Enterprise Media Manager or EMM server, this can be integrated into the Master, depending on your infra if it needs it to [I'm not going through definition of these so please do your part and read on other articles ;)].

Basically, all the schedules [and catalogs - valid images of a client, archiving, and restorations] are stored in the Master server. If I'm not mistaken, this is being handled by nbpem daemon which checks for backup due to all clients or makes a list for each policy that are due to run and submits it to nbjm. nbjm in turn, gets the necessary resource by coordinating with nbrb to start the backup. nbrb is responsible for allocation of resource to a job with the help of nbemm - holds the info about media and device configurations. Daemons discussed belongs to what we call Intelligent Resource Manager (IRM - resides in Master) who's working together with EMM to schedule and allocate resources for the job.

There are other daemons but I'll not go through each discussing each, however, here are some [take note of the version of your NBU, there might be some daemons that were replaced, so check out the docs]:

Master: bpcd, bpcompatd, bpdbm, ltid, nbnos, nbrd, nbsl, nbsvcmon, vmd, and pbx_exchange
Media Manager: bpcd, bpcompatd, ltid, nbnos, nbsl, nbsvcmon, vmd, pbx_exchange
Client: bpcd

Note that bpcd is the only daemon that runs on Client.
Check using:
$ netstat -an | grep -i bpcd
tcp 0 0 *.bpcd *.* LISTEN

Next up: IRM tells the Media Manager [handles the robot/tape silo] or MM to assign drive and gather data from the Client. So MM, requests the Client for the image of backup, which generates it, and sends back. Media Manager tells the Robotic Control [Robot Tape Library/Silo - I'm not sure it they're a single entity] to find a tape and mount it, afterwhich, MM sends the data to the tape.
****** END ******

That is the general picture of how backup is iniated and done. Again, I'd like to raise that these are my understanding, and I'm doing [when time permits] other reading and may modify this from time to time to correct info. But I always welcome comments: corrections, etc. which will benefit - hopefully - us all.

Note: This is not all my idea. I've read from other articles but forgot the url but thanks to whoever wrote and provided me such materials.

Filesystem extention - fsadm errno2

2008-10-18T11:52:00.009+08:00

Hi, This will be my first contribution for my friend's blog site. Well, This is just another urgent file system increase for other team restoration task. Basically what happen was after I extended the VG and about to extend the LV an error occurred indicating "vxfs fsadm: cannot open /oracle/A6C/mirrlogA/lost+found/.fsadm - errno 2".

The issue was resolved by recreating the lost+found then re-executing the fsadm.

[root@dagz:/oracle/A6C/origlogB]
# lvextend -L 20000 /dev/vg_A6C_00/lv_mirrlogA
Logical volume "/dev/vg_A6C_00/lv_mirrlogA" has been successfully extended.
Volume Group configuration for /dev/vg_A6C_00 has been saved in /etc/lvmconf/vg_A6C_00.conf

[root@dagz:/oracle/A6C/origlogB]
# fsadm -b 20000m /oracle/A6C/mirrlogA
fsadm: /etc/default/fs is used for determining the file system type
vxfs fsadm: cannot open /oracle/A6C/mirrlogA/lost+found/.fsadm - errno 2

[root@dagz:/oracle/A6C/origlogB]
# cd /oracle/A6C/mirrlogA

[root@dagz:/oracle/A6C/mirrlogA]
# mklost+found
creating slots...
removing dummy files...
done
drwxr-xr-x 2 root sys 4096 Oct 15 10:59 /oracle/A6C/mirrlogA/lost+found

[root@dagz:/oracle/A6C/mirrlogA]
# fsadm -b 20000m /oracle/A6C/mirrlogA
fsadm: /etc/default/fs is used for determining the file system type
fsadm: /dev/vg_A6C_00/rlv_mirrlogA is currently 5120000 sectors - size will be increased

[root@dagz:/oracle/A6C/mirrlogA]
# bdf /oracle/A6C/mirrlogA
Filesystem kbytes used avail %used Mounted on
/dev/vg_A6C_00/lv_mirrlogA
20480000 4102261 15354200 21% /oracle/A6C/mirrlogA

Unicenter Autosys: job scheduling tool

2008-09-29T14:36:00.023+08:00

I was sad and excited, may be li'l bit happy, when I started with my new work. Sad for am going to miss colleagues, the Neptune project in whole, except the SAP team - no love lost, among others. Excited for a new challenge looms; I thought.

I was useless/worthless in my team for the whole month - FICC IT PTF, whatever that means; while the team comprises: Indian, Chinese, locals [Singaporean], and us - Pinoy. Lucky for us, one of the shift leads or station managers is a Pinoy; I call him The Director - I find the job bit boring. It may be because I still don't have the access on other applications but, it could be the job is - as my shift lead puts it - for kindergarten. Sorry, just quoting. It makes me miss my SYSSUPPORT access. But, of course, this doesn't mean that I'm gonna stop writing about UNIX. Nada! It's a continuous learning. Next target: Solaris and RHEL. We got books here, as well as, CBTs.

On the side note, I came to learn a new application. It's the Unicenter Autosys - a job scheduling tool. Just like the Control-M, the one that we use before. It's just like a cron job, with conditions that can be set in order for that job to run. As for my job, in our team, tickets are assigned via "tasks". You are assigned tasks to be done on certain time [with or without SLA]. So, everyone has got something to do. You can't run, you can't hide!

As for the Autosys, we use it to check on jobs if they run successfully or otherwise. As mentioned, jobs should meet certain conditions before it will be executed. Such conditions ranges from time to dependencies to other jobs. I'll post here next time some results from my tasks. As for the meantime, please read on Unicenter Autosys.

*************************************************************
The "as for now" is over. I "stole" some results from the tasks assigned to me today. You may have read about it but here is the example:

user101@server202> aj IRS_server203_EOD-EUR

Job Name Last Start Last End ST Run Pri/Xit
__________________ _____________ _____________ __ _______ ___

IRS_server203_EOD-EUR 09/30/2008 21:00 09/30/2008 21:11 SU 24773663/1
user101@server202> aj IRS_server203_EOD-EUR -q

/* ----------------- IRS_server203_EOD-EUR ----------------- */

insert_job: IRS_server203_EOD-EUR job_type: c
command: $$XXX/Scripts/Autosys/IRS_server203_EOD-EUR.sh
machine: server203
#owner: XXX
permission: gx,mx
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "21:00"
description: "Rerun=0;663;Chat : XXX ; DL-xxx-yyy , DL-zzz-aaa"
std_err_file: $$XXX/Scripts/Autosys/Logs/IRS_server203_EOD-EUR.err
min_run_alarm: 1
max_run_alarm: 10
alarm_if_fail: 1
timezone: US/Eastern
user101@server202> aj IRS_server203_EOD-EUR -r -1

Job Name Last Start Last End ST Run Pri/Xit
__________________ _____________ _____________ __ _______ ___

IRS_server203_EOD-EUR 09/29/2008 21:00 09/29/2008 21:13 SU 24728737/1
user101@server202>

You might wonder about the "aj" command, and you may not find it anywhere. Before chaos reigns in our midst, it's just an alias for "autorep -J".

Here, another example - a running job:

user101@server202> aj FI_FCTxx_DownStream_BOX
Job Name Last Start Last End ST Run Pri/Xit
_______________ _____________ ____________ __ _______ ___

FI_FCTxx_DownStream_BOX 09/30/2008 21:29 ----- RU 24767669/1
FI_FCTxx_Settlement 09/30/2008 21:29 ----- RU 24767669/1
FI_FCTxx_JetBase 09/30/2008 21:30 09/30/2008 21:31 SU 24767669/1
FI_FCTxx_JetBase_SFtp 09/11/2008 21:38 09/11/2008 21:38 OI 24115468/1
FI_FCTxx_Socrates 09/30/2008 21:29 09/30/2008 21:30 SU 24767669/1
FI_FCTxx_Socrates_SFtp 09/30/2008 21:30 09/30/2008 21:31 SU 24767669/1
FI_FCTxx_MRC 09/30/2008 21:29 09/30/2008 21:30 SU 24767669/1
FI_FCTxx_MRC_SFtp 09/30/2008 21:30 09/30/2008 21:31 SU 24767669/1
FI_FCTxx_CPVar_SFtp 09/30/2008 21:30 09/30/2008 21:31 SU 24767669/1

user101@server202> job_depends -c -J FI_FCTxx_Settlement
______________________________________________________
Start Dependent
Job Name Status Date Cond? Cond? Jobs?
-------- ------ --------------- ----- ---------
FI_FCTxx_Settlement RUNNING No No No
______________________________________________________

user101@server202> job_depends -d -J FI_FCTxx_Settlement
Job Dependency Report

Job Name Date Cond? Atomic Start Conditions
______________________ _______________ ________________

FI_FCTxx_Settlement ------- -------
user101@server202>

Microsoft’s take on virtualization: re-print edition

2008-09-29T09:04:00.005+08:00

By Anna Valmero
INQUIRER.net
First Posted 18:51:00 09/26/2008

MANILA, Philippines -- “The IT industry is in the middle of the next paradigm shift toward virtualization,” said Bill Hilf, general manager of Windows Server and Platform Strategy for Microsoft Corp. Hilf visited the Philippines Friday to promote the release of Microsoft’s virtualization software in the country.

Virtualization, as defined by Microsoft, means having the power to run multiple systems, such as server, storage device, network or even an operating system and applications in just one machine, making a business run its system more efficiently. This virtualization trend comes at the right time to harness the processing power of multi-core processors being deployed in the industry.

As the next trend to shape the IT industry, virtualization can revolutionize the use of computing machines, said Hilf.

A virtualized box can run multiple operating systems compared to a traditional server without virtualization that can handle only one operating system. By deploying virtual machines, one box can serve the function achieved previously by two or more boxes without virtualization. This is also true for desktops.

“Virtualization leads to a lot of savings,” said Hilf.

First, the savings can come from lesser space to be occupied by boxes with virtualization. By having lesser boxes, this also reduces the costs required to cool the server systems. Cooling is required to remove the heat from the server environment, which when reduced translates to lower power utilities. A company can also cut the number of servers it needs to acquire for its business since one box running virtualization can serve the function of two or more servers. Virtualization software packs also offer bulk licensing, as in the case of Microsoft’s premium software products, thus bringing down licensing costs.

Another advantage of virtualization is the ability of a single machine to handle previous-generation applications while running modern applications. This is specifically beneficial to businesses that maintain servers to run old legacy applications.

“By being able to run these old legacy applications on a new, better-performance server, it will help save a lot of money,” Hilf said. Organizations thus achieve better management efficiencies in its resources.

Aside from being able to run multiple operating systems on one desktop, how can the common business person benefit from virtualization?

Explains Hilf: “Imagine this: You have a small device, maybe a laptop or just a small box. Through virtualization, you are able to access via streaming your required applications from a remote server. Technology such as Microsoft's Remote Display Protocol streams keystrokes and mouse clicks to the server and a view of the screen back to the user, who can be located hundreds of miles away from the server or desktop.”

Simply, it can allow users to access a remote PC when he is offsite.

Link: Microsoft’s take on virtualization.

***
I was surprised when I saw and read this article. What the *&%$?!? This is already an "old" [relatively] thing. You should check and get VMWare instead. Well, what's the connection with HP-UX in particular? This can be done on the hardware level: node partitioning [npar]; 'though there is such as virtual partitioning [vpar].

Fibre channel host port is OFFLINE

2008-08-18T23:33:00.011+08:00

This may not be your regular error but this is quite useful especially for those who have SAN implementation in their infra [SC21948063]. When we did receive a call from the customer[?], we were compelled to investigate on the raised issue. Basically, I myself, ain't familiar with this. But since this calls for investigation, I accepted the ticket without second thought. Ahhh, ticket! Well I checked first if the box went on a maintenance. So, I did: `uptime`, `date`, `who -r`. Then I came to check for the syslog [HP-UX]; and HW scan [`ioscan -fnCfc`]. Well [why do I use this word too often?!? Well, I don't know either!], there are interesting entries that I found.

[root@server001:/home/user001]
# uptime
11:22am up 323 days, 15:45, 3 users, load average: 0.51, 0.37, 0.31

[root@server001:/home/user001]
# date
Mon Aug 18 11:22:05 EDT 2008

[root@server001:/home/user001]
# who -r
. run-level 4 Sep 29 19:37 4 0 S

[root@server001:/root]
# cat /home/user001/fc_20080818.log
Info for /dev/fcd0

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x454a00
Previous N_Port_id is = 0x454a00
N_Port Node World Wide Name = 0x50060b00003966a5
N_Port Port World Wide Name = 0x50060b00003966a4
Switch Port World Wide Name = 0x204a006069e2147e
Switch Node World Wide Name = 0x1000006069e2147e
Driver state = ONLINE
Hardware Path is = 0/2/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd1

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a7
N_Port Port World Wide Name = 0x50060b00003966a6
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/2/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd2

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x464a00
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a9
N_Port Port World Wide Name = 0x50060b00003966a8
Switch Port World Wide Name = 0x204a006069e2145e
Switch Node World Wide Name = 0x1000006069e2145e
Driver state = ONLINE
Hardware Path is = 0/4/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd3

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966ab
N_Port Port World Wide Name = 0x50060b00003966aa
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/4/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd4

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396885
N_Port Port World Wide Name = 0x50060b0000396884
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/5/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd5

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x686900
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396887
N_Port Port World Wide Name = 0x50060b0000396886
Switch Port World Wide Name = 0x2069006069e207b2
Switch Node World Wide Name = 0x1000006069e207b2
Driver state = ONLINE
Hardware Path is = 0/5/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

[root@server001:/home/user001]
# cat syslog.out
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan

[root@server001:/home/user001]
#

If you can see from the logs the line that contains this: Driver state = AWAITING_LINK_UP. As per suggestion [which resolved the issue btw] from http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1219074300410+28353475&threadId=1165601, this could be a port issue:

The "AWAITING_LINK_UP" state describes the problem. Check with your SAN team about the status of the switch port (F-port) to which the HBA is connected.

To which the caller affirm that the FC was improperly connected. But still, there could be some other reason for this. I'll wait for the response/action...

Also, if you can see from the syslog, you can run this for more info:

[root@server001:/home/user001]
# /opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a

FTP and IP address

2008-08-06T10:41:00.015+08:00

Two issues that struck me today: something to do with FTP and the other, IP address.

1. FTP: A requestor is asking if the files transferred from a source regardless of the permissions will be "converted" to the permissions as defined in the destination server/location. Which led me to some digging.

So basically, FTP permissions can be set. But not without affecting all users, i.e., globally as this is done via inetd. Or you can check these:

A preview:

Sridhar Bhaskarla May 26, 2004 18:47:14 GMT
--------------------------------------------------------------------------------
Hi,
Set the umask in the ftp session itself. For ex.,
ftp>umask 022

ftp>put file1 file2
-Sri

Griselda Sanchez May 26, 2004 18:59:24 GMT
--------------------------------------------------------------------------------
Thanks Sri for your advice, but Is there any way to apply this permanent?

Mel Burslan May 26, 2004 19:04:19 GMT
--------------------------------------------------------------------------------
other than what Sri said, you can not set some ftp sessions' umask to something while keeping the others at something else.
in /etc/inetd.conf, you will find a line similar to this,

ftp stream tcp nowait root /usr/lbin/ftpd ftpd -L 002

where the 002 at the tail end, is the default umask for files put to the server via ftp. you can change this value to set the umask but it will effect every user.

Michael Tully May 26, 2004 19:06:46 GMT
--------------------------------------------------------------------------------
There is no way to set it permanently, other than enforcing these things from within a script wrapper.

Sridhar Bhaskarla May 26, 2004 19:26:20 GMT
--------------------------------------------------------------------------------
Hi,
As far as I know, you can do it either for all the users by configuring ftpd in inetd.conf with -u option or for a directory using ftpaccess file.
Look at 'man ftpaccess' on directory level permissions. Search for "upload".
-Sri

Bill Hassell May 26, 2004 22:15:26 GMT
--------------------------------------------------------------------------------
There is no way to change this for a single user. ftpaccess can allow or disallow the use of umask, but not preset the mask. You'll need to script the ftp transfer and includxe the required permission changes in your script. This is easy in Unix, a real pain with Macs and PCs, especially if the end user insists on using a fancy GUI interface.

But, I guess, it would much better to set things within the source thru umask [set it via .profile].

2. IP address [SC#21810036]: This has to be done on a Solaris box. An IP - virtual - (although two were requested), needs to be "un-bind" from the NIC. Here are the some of the info I got:

[root@server:/root]
# ifconfig -a ["pipe"] grep -e 192.44.122.154 -e 192.44.122.226
....
hme0:26: flags=1000842 mtu 1500 index 2
inet 192.44.122.226 netmask fffffe00 broadcast 192.44.123.255
....
[root@server4013:/root]
# ping 192.44.122.226
no answer from 192.44.122.226

[root@server4013:/root]
# netstat -r

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
192.44.123.0 server1.com U 1 84389 hme0:4
6192.44.123.0 server2.com U 1 0 hme0:47
192.44.123.0 server3.com U 1 0 hme0:48
192.44.123.0 server4.com U 1 0 hme0:49
192.44.123.0 192.44.123.79 U 1 0 hme0:50
192.44.123.0 server5.com U 1 0 hme0:51
192.44.122.0 server4013.com U 1 940909 hme0
.............
default 192.44.122.4 UG 11260332localhost localhost UH 417021843 lo0

Routing Table: IPv6
Destination/Mask Gateway Flags Ref Use If
--------------------------- --------------------------- ----- --- ------ -----
fe80::/10 fe80::a00:20ff:fef9:81d4 U 1 0 hme0
ff00::/8 fe80::a00:20ff:fef9:81d4 U 1 0 hme0
default fe80::a00:20ff:fef9:81d4 U 1 0 hme0
localhost localhost UH 1 1736 lo0

[root@server4013:/root]
#

I also got some info from this site: http://www.tek-tips.com/faqs.cfm?fid=4083

But how?!?

The answer: It was a futile effort. Nothing much to read about.

"only one Interface was there : 192.44.122.226
removed the interface"

What can get from this?!? SO, I have to know this by hook or by crook or by research. And I found this: http://www.experts-exchange.com/OS/Unix/Solaris/Q_21713340.html

jephilc:

Hi

1) There are two ways to remove logical interfaces...
Either do this for each one you want to remove (as root)

# ifconfig hme0:1 down unplumb

or if you want to specify the address you can do this:

# ifconfig hme0 removeif 172.16.17.241

[ The above shows two ways of removing the 241 address.

2) To see what type of network interface you've got for example, for an ultra system run this as root

# /usr/platform/sun4u/sbin/prtdiag -v

You'll see a line like this for an hme interface

0 PCI-1 33 1 network-SUNW,hme

Check dmesg for messages after a boot which will tell you the speed the interface initialized and whether it was full or half duplex etc.

3) Use dmesg or look in /var/adm/messages to tell you at what speed and duplex setting the interface is running at

4) To hardcode the interface at a specific speed and duplex, you use the ndd command as root, but most of the time the interface will probably be set to autonegotiate. The autonegotiate facility is extremely useful especially if you have legacy systems on your network that might be running at 10 Mbps as your system will automatically talk to them OK. If you turn off this facility, and you are running legacy network interfaces, you could lose connectivity to them or cause errors, so it would be wise to check this first.

First check whether the general autoneg capability is set to 1 by running

# ndd /dev/hme adv_autoneg_cap

If it's set to 1 then you need to turn it off by doing:

# ndd -set /dev/hme adv_autoneg_cap 0

then if you want to manually set it to 100 full duplex run this:

# ndd -set /dev/hme adv_100fdx_cap 1

and make sure the other settings are set to 0. See the other available settings by listing the possible variables with this command:

# ndd /dev/hme \?

The ones marked (read and write) are the ones you can set.

Good luck

John

01.28.2006 at 02:45PM PST, ID: 15814641

jephilc:

Hi again

Apologies but I forgot to mention the /etc/hostname.hme0 file.

If the entries had been made permanent for the logical interfaces, then you will need to edit this file to remove the relevant interface entries. This will be needed so that the interfaces are permanently removed and do not re-appear when you reboot. In my original posting, I was assuming that you wanted to do the operation on the fly.

If rebooting isn't a problem, then all you really need to do in order to remove the interfaces is to edit /etc/hostname.hme0, remove the entries and reboot.

Hope this helps

John

Well, I hope did...

LUN: POWERFAILED

2008-07-21T11:49:00.010+08:00

I did encounter this error eons ago but never really had the chance to write about it.

Anyway, I'll be using some inputs provided by my colleagues. Please take note that this is merely for conversion or identifying the disk that have a power failure.

From the /var/adm/syslog.log:
...
Jul 20 21:54:50 server4385 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000048753840), from raw device 0x1f060100 (with priority: 0, and current flags: 0x40) to raw device 0x1f078100 (with priority: 1, and current flags: 0x0).
Jul 20 21:54:50 server4385 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000048753840), from raw device 0x1f060100 (with priority: 0, and current flags: 0x40) to raw device 0x1f078100 (with priority: 1, and current flags: 0x0).
Jul 20 21:54:50 server4385 vmunix: LVM: Restored PV 1 to VG 1.
Jul 20 21:54:50 server4385 vmunix: LVM: Restored PV 1 to VG 1.
Jul 20 21:54:54 server385 vmunix: LVM: vg[1]: pvnum=1 (dev_t=0x1f078100) is POWERFAILED
Jul 20 21:54:54 server4385 vmunix: LVM: vg[1]: pvnum=1 (dev_t=0x1f078100) is POWERFAILED
Jul 20 21:55:04 server4385 vmunix: LVM: Recovered Path (device 0x1f060100) to PV 1 in VG 1.
Jul 20 21:55:04 server4385 vmunix: LVM: Recovered Path (device 0x1f060100) to PV 1 in VG 1.
Jul 20 21:55:04 server4385 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000048753840), from raw device 0x1f078100 (with priority: 1, and current flags: 0x0) to raw device 0x1f060100 (with priority: 0, and current flags: 0x80).
Jul 20 21:55:04 server4385 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000048753840), from raw device 0x1f078100 (with priority: 1, and current flags: 0x0) to raw device 0x1f060100 (with priority: 0, and current flags: 0x80).
...

We'll use the entry "dev_t=0x1f078100". To convert to the exact disk, take the last six (6) digits, e.g., 078100, and check from /dev/dsk:

# ll /dev/dsk | grep 078100
brw-r----- 1 bin sys 31 0x078100 Apr 19 23:47 c7t8d1

Now, this represents the device file for the disk. Since this is being used for LVM, we can use pvdisplay, vgdisplay, or lvdisplay to check partly on the status of the data written on it.

VxVM - Solaris

2008-07-21T10:41:00.004+08:00

Well, well, well. What have we got here?! A FS extension for a Solaris box. Hmmm, okay! Let me check some old notes.

Nothing fancy really. Just like any ordinary request but got to consider some rules since it involves rootdg, which is a bit sensitive. Every move must be evaluated. Remember: it's a root disk!

[root@server254:/etc/vx/bin]
# vxdg list
NAME STATE ID
rootdg enabled 1052259816.1025.server254

[root@server254:/etc/vx/bin]
# vxassist -g rootdg maxsize
Maximum volume size: 8192 (4Mb)

The above output shows free space of around 32GB for the dg.

To extend [m after 150 signifies the value in MB]:

[root@server254:/etc/vx/bin]
# /etc/vx/bin/vxresize -g rootdg volume_name 150m

Anyway, I found this useful link for UNIX administration: http://www.hyborian.demon.co.uk/notes/

Ignite error anew

2008-07-17T11:24:00.008+08:00

Chapter 19. After Antonia received a beating from a Brigatisti, Beowulf Agate saved and brought her to a doctor. I was reading this novel when Maki asked if I encountered the error below:

[root@server21:/opt/ignite/bin]
# /opt/ignite/bin/make_net_recovery -s ignite_server -x inc_entire=vg00
* Creating NFS mount directories for configuration files.
======= 07/16/08 23:12:49 EDT Started /opt/ignite/bin/make_net_recovery. (Wed
Jul 16 23:12:49 EDT 2008)
@(#)Ignite-UX Revision C.7.4.157
@(#)ignite/net_recovery (opt) Revision:
/branches/IUX_RA0712/ignite/src@72068 Last Modified: 2007-11-01 14:16:06 -0600 (Thu, 01 Nov 2007)

* Testing for necessary pax patch.
* Checking Versions of Recovery Tools
* Creating System Configuration.
* /opt/ignite/bin/save_config -f /var/opt/ignite/recovery/client_mnt/0x0
00E7FED2223/recovery/2008-07-16,23:12
/system_cfg vg00/opt/ignite/bin/save_config[16]: lssf: Execute permission denied.
save_config: Error - unknown disk type for /dev/dsk/c0t6d0s2, not SCSI or HPFL
awk: Cannot find or open file /var/tmp/swapinfo.tmp.
The source line number is 1.
awk: Cannot find or open file /var/tmp/swapinfo.tmp.
The source line number is 1.
save_config: Error - cannot determine primary swap size
ERROR: /opt/ignite/bin/save_config failed
======= 07/16/08 23:13:26 EDT make_net_recovery completed unsuccessfully

[root@server21:/opt/ignite/bin]
#

Well the first course of action is to check on the execution of `lssf` [which btw, resolved the issue - check with the permission or file rule on this: "SEC team"]. However, just out of curiosity, I checked on the swap. It showed 3 but upon checking on the /etc/fstab entries, it listed only 2.?. So, here comes the experiment, I added the missing and activated it. Then tried to run the ignite backup again. But, to no avail. Still getting the same error. Well, see, I learned.

NFS

2008-07-14T14:47:00.005+08:00

I thought it was just another NFS request. Well, technically, it is. What I meant was another mount, and there it goes. But, I was wrong [again]! Just like the other aspect of my life. Hmm, forget it!

Ok. Let's check on the request.

Request: Mount /var/opt/edis/idocs/directory1 to server2 from box1.

Simple? May be.

Steps:
1. Entry was added to exports
2. run `exportfs -a` but error was generated:
parent directory already exported

Several attempts [solution] were tried but none succeeded.
1. comment the parent directory
2. manually exporting the child directory

A colleague thought of a brilliant idea. It was a dull day so I never bother to explore such. Anyway, he suggested to add the client to the parent directory's list, as well as make a root entry for it, then export the FS. Successful, at least the initial part.

Login to the client side. Made a copy of the fstab [a great practice, just in case a stupid me does a stupid thing]. Then add it as:

# cp -p /etc/fstab /etc/fstab.backup.`date`
# vi /etc/fstab
...
/var/opt/edis/idocs/directory1 /var/opt/edis/idocs/directory1 nfs rw ...
...
~
~

The entry added for the client was specific. It means, although the "whole" parent directory was exported, I think it is possible to just put the specific directory to be accessed. At least for this case. I think, no! I need to review more about NFS. Adios!

Reboot after panic: Data page fault

2008-06-29T08:43:00.007+08:00

One of the servers that we monitor rebooted on a panic. The dumps, I think, will be submitted to the HPRC for decoding what caused the panic. But, for DPF's, it is most unlikely that it was caused by hardware failure. Probably, some application passed something on the kernel that it didn't know how to process it. Anyway, just an opinion. I'm not sure if I can get anything from the DTS. A root cause analysis is needed for this.

Here are the files on the crash dumps.

[root@box1:/var/adm/crash/crash.0]
# ll
total 2113190
-rw-r--r-- 1 root root 1556 Jun 28 19:05 INDEX
-rw-r--r-- 1 root root 281784 Jun 28 19:02 SEOS
-rw-r--r-- 1 root root 134189056 Jun 28 19:02 image.1.1
-rw-r--r-- 1 root root 134197248 Jun 28 19:02 image.1.2
-rw-r--r-- 1 root root 134152192 Jun 28 19:03 image.1.3
-rw-r--r-- 1 root root 89419776 Jun 28 19:03 image.1.4
-rw-r--r-- 1 root root 134180864 Jun 28 19:04 image.2.1
-rw-r--r-- 1 root root 134180864 Jun 28 19:04 image.2.2
-rw-r--r-- 1 root root 134193152 Jun 28 19:04 image.2.3
-rw-r--r-- 1 root root 134168576 Jun 28 19:05 image.2.4
-rw-r--r-- 1 root root 36978688 Jun 28 19:05 image.2.5
-rw-r--r-- 1 root root 16007272 Jun 28 19:02 vmunix

[root@box1:/var/adm/crash/crash.0]
# more INDEX
comment savecrash crash dump INDEX file
version 2
hostname box1
modelname 9000/800/N4000-44
panic Data page fault
dumptime 1214693427 Sat Jun 28 18:50:27 EDT 2008
savetime 1214694120 Sat Jun 28 19:02:00 EDT 2008
release @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 buil
d' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
memsize 4294967296
chunksize 134217728
module /stand/vmunix vmunix 16007272 3341476060
module /stand/dlkm/mod.d/SEOS SEOS 281784 3144992042
image image.1.1 0x0000000000000000 0x0000000007ff9000 0x0000000000000000 0x0000000000008987 356976270
image image.1.2 0x0000000000000000 0x0000000007ffb000 0x0000000000008988 0x0000000000015447 637338283
image image.1.3 0x0000000000000000 0x0000000007ff0000 0x0000000000015448 0x0000000000066d7f 35130470
image image.1.4 0x0000000000000000 0x0000000005547000 0x0000000000066d80 0x000000000007ffff 3529390633
image image.2.1 0x0000000000000000 0x0000000007ff7000 0x0000000000180000 0x00000000001a7307 719265648
image image.2.2 0x0000000000000000 0x0000000007ff7000 0x00000000001a7308 0x00000000001bec17 3529725656
image image.2.3 0x0000000000000000 0x0000000007ffa000 0x00000000001bec18 0x00000000001ca9df 560273249
image image.2.4 0x0000000000000000 0x0000000007ff4000 0x00000000001ca9e0 0x00000000001f5e0f 3332528375
image image.2.5 0x0000000000000000 0x0000000002344000 0x00000000001f5e10 0x00000000001fffff 3748535493

[root@box1:/var/adm/crash/crash.0]
# uptime
8:29pm up 1:28, 2 users, load average: 0.14, 0.18, 0.11

[root@box1:/var/adm/crash/crash.0]
# date
Sat Jun 28 20:29:31 EDT 2008

[root@box1:/var/adm/crash/crash.0]
# more /etc/shutdownlog
10:02 Mon Oct 10, 2005. Reboot:
11:22 Mon Oct 10, 2005. Reboot: (by SAM)
11:25 Mon Oct 10, 2005. Reboot: (by bdhp4420!root)
06:16 Tue Oct 11, 2005. Reboot: (by bdhp4420!root)
09:56 Thu Oct 20, 2005. Reboot: (by bdhp4420!root)
10:06 Thu Apr 20, 2006. Reboot: (by SAM)
10:12 Thu Apr 20, 2006. Reboot: (by bdhp4420!root)
19:05 Sat Jun 28 2008. Reboot after panic: Data page fault

[root@box1:/var/tombstones]
# ll -rt
total 252
-rw-r--r-- 1 root root 14720 Oct 7 2005 ts93
-rw-r--r-- 1 root root 14720 Oct 10 2005 ts94
-rw-r--r-- 1 root root 14720 Oct 10 2005 ts95
-rw-r--r-- 1 root root 14720 Oct 11 2005 ts96
-rw-r--r-- 1 root root 14720 Oct 20 2005 ts97
-rw-r--r-- 1 root root 14720 Apr 20 2006 ts98
-rw-r--r-- 1 root root 14720 Jun 28 19:02 ts99
-rw-r--r-- 1 root root 20873 Jun 28 19:08 cpumap

[root@box1:/var/tombstones]
#

On the side note, some application processes are not running. I saw earlier that one of them, Oracle, was started manually. So, I guess the same goes with the rest.

UPDATE: The team already opened an HPRC case for this and at the same time, a Problem Record (PR#54269).

Reconfiguring an HP-UX (11iv1) kernel

2008-05-25T12:05:00.023+08:00

It's whole new experience [heaven]. I thought it will remain a wish. Performing this is a long-shot in our working environment. But, what could be sweeter than a wish coming true? Nada! Of course, other than.. nah, nevermind! Ok, we have a project to have a kernel parameter changed. Upon doing a prework, we have this initial value:

[root@hpux09:/stand/build]
# kmtune -q vx_maxlink
Parameter Current Dyn Planned Module Version
==========================================
vx_maxlink 32767 - 32767

[root@hpux05:/stand/build]

We usually perform these kind of changes via SAM. But, wait! What the f*%$? Where's the vx_maxlink parameter?!? "Uhmm, hey [referring to my colleague], can you check it with L3?" So that's it. There are some parameters that does not show [or linked] on SAM. Changes to be made are to be performed via CLI. So, here are the steps I followed:

[root@hpux09:/stand/build]
# /usr/lbin/sysadm/system_prep -s system

[root@hpux09:/stand/build]
# kmtune -q vx_maxlink
Parameter Current Dyn Planned Module Version
==========================================
vx_maxlink 32767 - 32767

[root@hpux09:/stand/build]
# kmtune -s vx_maxlink=65534

[root@hpux09:/stand/build]
# kmtune -q vx_maxlink
Parameter Current Dyn Planned Module Version
==========================================
vx_maxlink 32767 - 65534

[root@hpux09:/stand/build]
# which mk_kernel
/usr/sbin/mk_kernel

[root@hpux09:/stand/build]
# mk_kernel -s system
Generating module: krm...
Generating module: SEOS...
Compiling conf.c...
Loading the kernel...
Generating kernel symbol table...

[root@hpux09:/stand/build]
# kmupdate

Kernel update request is scheduled.

Default kernel /stand/vmunix will be updated by
newly built kernel /stand/build/vmunix_test
at next system shutdown or startup time.

[root@hpux09:/stand/build]
# shutdown -ry 0
Shutdown cannot be run from a mounted file system -- exiting shutdown.
Change directories to the root volume ("/" will work) and try again.

[root@hpux09:/stand/build]
# cd /

[root@hpux09:/]
# shutdown -ry 0

SHUTDOWN PROGRAM
05/24/08 23:40:31 EDT

Broadcast Message from root (pts/5) Sat May 24 23:40:31...
PLEASE LOG OFF NOW ! ! !
System maintenance about to begin.
All processes will be terminated in 0 seconds.

Broadcast Message from root (pts/5) Sat May 24 23:40:31...
SYSTEM BEING BROUGHT DOWN NOW ! ! !

/sbin/auto_parms: DHCP access is disabled (see /etc/auto_parms.log)

For now, we have to wait for box to come up and check if the change we applied took effect. [Cross-finger] Hoping it did.?! It's driving me crazy [and very excited!].

What the f*%$ have I done?? I am doomed! The kernel parameter didn't change at all. The value is still under planned. My heart raced and pounded. Hey! I'm no superman. Looking for a reason to have the window time extended. Deym! [Temporary, still to hear a LOT about this during our weekly meeting] Fortunate for me (?), the change was so important that it left no choice for the application team, requestor, and box owner to extend the time and allow me to give it another go. But, this time? I got L3's attention! I consulted them, and they gave me an SOP [btw, for the record, of which I'm not aware of and was not provided]. [Another] But, to make sure, I let the L3 do it [I got my hands tied already, so I'm not taking any chances - not now but, definitely will love to do it again, anytime, anywhere!], and check how he did it a bit later. A few, very long, minutes later, he's working his magic. And here's how:

cd /stand/build
ll system [optional but essential]
kmtune -q vx_maxlink
/usr/lbin/sysadm/system_prep -s system
kmtune -s vx_maxlink=65534 -S ./system # This is what I missed; writing to system file
/usr/sbin/mk_kernel -s system
kmupdate
cd ..
cp -p system system_prev
mv build/system .
kmtune -q vx_maxlink
[now all I need is to reboot the box, and done!]

Well, folks, I hope you learned new. For me? I learned a TON!

And oh, make sure to watch out for the /stand FS getting full. You might end up just like it. I tell you, it's nasty. May be giving a system_prep will clear it... Well, just a thought. Good luck to us all.

This was added a bit later [July something of 2008].

Here's the procedure for HPUX 11.23: (explanation? Later)

# mv /stand/system /stand/system.orig

# kconfig -e /stand/system

CEdit /stand/system file and remove all "Tunable Parameters".
Copy and paste Tunable parameters ( lined between START KERNEL PARM and END KERNEL PARM) from /tmp/logfile.
Save and exit.

# kconfig -i /stand/system

# shutdown -r 0