Mirroring a Root Volume or Disk, and /boot

I slept late last night, or may be late this early morning. Does it make sense? Anyway, when I woke up, I prepared my food. Forget it. Alright, here we go.

I was doing my morning ritual - if got no work - checking my mails, visiting PACLAND, NBA, going thru today's headlines including stock market? Freakin' sick, ayt?! Can't help it esp. in times like these and working in a bank!

Whew! I find it funny writing about this. Another problem is, I was worrying more on the title than the content.

Seriously, here it is.

I was reading RHEL 5 Deployment Guide. Actually, I was continuing. I was considering of skipping the LVM part but decided to go for it anyway. And something, which triggered this blog, caught my attention: The /boot/ partition cannot reside on an LVM volume because the GRUB boot loader cannot read it.

Interesting point. I admit, I was more of an HP-UX LVM guy. Shame on me for the confusion. I begun to ask a former colleague - Che, kudos - to execute some commands for us to verify if it the same with HP-UX. I was so pissed at myself and at the same time wanted to laugh that I was not sure which command to say. Finally, I got me a `setboot`:

[root@server1032:/root]
# setboot
Primary bootpath : 0/1/1/0.0.0
Alternate bootpath : 0/1/1/0.1.0

Autoboot is ON (enabled)
Autosearch is ON (enabled)

Which at that point, doesn't answer the question nor mean anything [yet]?!. So, I Googled it with key words: HP-UX LVM /boot. See? /boot in HP-UX? Another evidence that I wasn't thinking right - since this is an HP-UX issue, I should have gone to ITRC.

Too, late.

... and there it is [as I end up with ITRC]. How can I forget it? It's included in vg00. In fact, there are commands such as mkboot and -B for pvcreate. And please take note of lvlnboot, there is this line that says: Boot.

On the side note, for a bad boot, mirrored disk replacement case, here's how it is done [PA-RISC and Integrity] - this is available on HP's docs:

---
Mirroring the Root Volume on PA-RISC Servers

To set up a mirrored root configuration, you must add a disk to the root volume group, mirror all the root logical volumes onto it, and make it bootable. For this example, the disk is at path 2/0/7.15.0 and has device special files named /dev/rdsk/c2t15d0 and /dev/dsk/c2t15d0.

1. Use the insf command with the -e option to make sure the device files are in place. For example:

# insf -e -H 2/0/7.15.0

You should now have the following device files for this disk:
/dev/dsk/c2t15d0 The entire disk (block access)
/dev/rdsk/c2t15d0 The entire disk (character access)

2. Create a physical volume using pvcreate with the -B option:

# pvcreate -B /dev/rdsk/c2t15d0

3. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t15d0

4. Use the mkboot command to set up the boot area:

# mkboot /dev/rdsk/c2t15d0

5. Use the mkboot command to add an autoboot file to the disk boot area. If you expect to boot from this disk only when you lose quorum, you can use the alternate string “hpux –lq” to disable quorum checking:

# mkboot –a “hpux” /dev/rdsk/c2t15d0

1. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t6d0 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00038 current /dev/vg00/lvol2 00000
00550 current /dev/vg00/lvol3 00000
00583 current /dev/vg00/lvol4 00000
00608 current /dev/vg00/lvol5 00000
00611 current /dev/vg00/lvol6 00000
00923 current /dev/vg00/lvol7 00000
01252 current /dev/vg00/lvol8 00000

In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t15d0
# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t15d0

6. Update the root volume group information:

# lvlnboot -R /dev/vg00

7. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

8. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 2/0/7.15.0

9. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t15d0

where l denotes LVM.
---

Mirroring the Root Volume on Integrity Servers

The procedure to mirror the root disk on Integrity servers is similar to the procedure for PA-RISC servers. The difference is that Integrity server boot disks are partitioned; you must set up the partitions, copy utilities to the EFI partition, and use the HP-UX partition device files for LVM commands.

For this example, the disk is at hardware path 0/1/1/0.1.0, with a device special file named /dev/rdsk/c2t1d0.

1. Partition the disk using the idisk command and a partition description file.

a. Create a partition description file. For example:

# vi /tmp/pdf

In this example the partition description file contains:
3
EFI 500MB
HPUX 100%
HPSP 400MB

b. Partition the disk using idisk and the partition description file created in step 1a:

# idisk -f /tmp/pdf -w /dev/rdsk/c2t1d0

To verify that your partitions are correctly laid out, run the following command:

# idisk /dev/rdsk/c2t1d0

2. Use the insf command with the -e option to create the device files for all the partitions. For example:

# insf -e -H 0/1/1/0.1.0

You should now have the following device files for this disk:
/dev/dsk/c2t1d0 The entire disk (block access)
/dev/rdsk/c2t1d0 The entire disk (character access)
/dev/dsk/c2t1d0s1 The EFI partition (block access)
/dev/rdsk/c2t1d0s1 The EFI partition (character access)
/dev/dsk/c2t1d0s2 The HP-UX partition (block access)
/dev/rdsk/c2t1d0s2 The HP-UX partition (character access)
/dev/dsk/c2t1d0s3 The Service partition (block access)
/dev/rdsk/c2t1d0s3 The Service partition (character access)

3. Create a physical volume using pvcreate with the -B option. Be sure to use the device file denoting the HP-UX partition.

# pvcreate -B /dev/rdsk/c2t1d0s2

4. Add the physical volume to your existing root volume group using vgextend:

# vgextend /dev/vg00 /dev/dsk/c2t1d0s2

5. Use the mkboot command to set up the boot area. Specify the -e and -l options to copy EFI utilities to the EFI partition, and use the device special file for the entire disk:

# mkboot –e –l /dev/rdsk/c2t1d0

6. Update the autoboot file in the EFI partition.

a. Create an AUTO file in the current directory. If you expect to boot from this disk only when you have lost quorum, you can use the alternate string “boot vmunix –lq” to disable quorum checking:

# echo “boot vmunix” > ./AUTO

b. Copy the file from the current directory into the new disk EFI partition. Make sure to use the device file with the s1 suffix:

# efi_cp -d /dev/rdsk/c2t1d0s1 ./AUTO /efi/hpux/auto

7. Use the lvextend command to mirror each logical volume in vg00 (the root volume group) onto the specified physical volume. The logical volumes must be extended in the same order that they are configured on the original boot disk. Use the pvdisplay command with the -v option to determine the list of logical volumes and their order. For example:

# pvdisplay -v /dev/dsk/c0t0d0s2 | grep 'current.*0000 $'
00000 current /dev/vg00/lvol1 00000
00010 current /dev/vg00/lvol2 00000
00138 current /dev/vg00/lvol3 00000
00151 current /dev/vg00/lvol4 00000
00158 current /dev/vg00/lvol5 00000
00159 current /dev/vg00/lvol6 00000
00271 current /dev/vg00/lvol7 00000
00408 current /dev/vg00/lvol8 00000

In this example, mirror the logical volumes as follows:
# lvextend –m 1 /dev/vg00/lvol1 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol2 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol3 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol4 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol5 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol6 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol7 /dev/dsk/c2t1d0s2
# lvextend –m 1 /dev/vg00/lvol8 /dev/dsk/c2t1d0s2

8. Update the root volume group information:

# lvlnboot -R /dev/vg00

9. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks:

# lvlnboot –v

10. Specify the mirror disk as the alternate boot path in nonvolatile memory:

# setboot –a 0/1/1/0.1.0

11. Add a line to /stand/bootconf for the new boot disk using vi or another text editor:

# vi /stand/bootconf
l /dev/dsk/c2t1d0s2

where l denotes LVM.
---

My first question has been answered. But, as a former S.A. and UNIX support engineer [for a huge infra], a redundant connection and/or backup is always in the back of our minds. As they say, "He who laughs last, has a backup."

So, the second question is, if /boot is not under the control of LVM, what is a good way to have a redundant connection? I don't have enough experience with this so, I turn to an old friend who has or had. We talked of RAID: software and hardware. For this case, he suggested RAIDS: 1, 5, or 1+0. In case you're server have a RAID controller, it is recommended to go for hardware. This and backup, he said, is enough to get you through.

Basically, 100MB is enough for /boot so, the rest of the disk, in case you decide to use LVM, can be brought under its control. While some others separate / as well for faster recovery process, he continued, for there were cases where the root VG, that contains the default file systems, is not detected. And this is a lengthy process of recovering it. It's a big no-no in critical systems, where downtime is associated with business loss [read: money].

So, I guess, that's it. I'm quite satisfied with the answers, and then some.

UPDATED: Veritas NetBackup Tutorial: some troubleshooting commands, etc.

Supposedly, I'll let several days to pass by before making this follow up but I was thinking, it could turn into weeks or even months before I can find time. So, before that happens, I'm going to publish the sequel this early. I don't know but, at times, no, often, I easily forget the details about things BUT it won't take time before I remember it anyway. Just a bit of focus and re-channeling of some energies and time and am back on track again. Ok, 'nuff with the gibberish.. moving forward...

Very early today, I laid down the basic config and executable files needed by NBU to run, at least in a UNIX environment. What's next are some basic troubleshooting commands that you might find useful in finding errors [code] generated by failure, or determine the culprit behind it.

And before we proceed any further, you may find it missing, i.e. topic about installation. As of this writing, I don't have a background in NBU installation. My experience is mostly on the maintenance part. So, please spare me of such questions. But just like any eager student, I'm always willing to learn and craves for knowledge.. at times!

Back!

I'll try to group these according to scenarios when they're needed but no guarantees.

# bpadm = text-based user interface; much quicker than jnbSA [Java-based]; can be used for restoration

# vmoprcmd -d = list of tape drives that are busy with active jobs; status [TLD, TLD-DOWN, AVR, PEND-TLD]; tapes mounted on to drive

# vmoprcmd -hoststatus -h hostname = should get similar o/p as below

Host `hostname` is ACTIVE

# vmoprcmd -h hostname -dps = will report if drive is SHARED; status should be UP

# vmdareq -driveinfo = check which drive is RESERVED or AVAILABLE

# tpconfig -d = tape device files configured for the machine and its status; helpful in identifying EMM

# robtest = to run SCSI pass-through commands; scans drive within library to find empty/full slot IDs; unloading and/or moving tapes across

# bptestbpcd -host hostname [-client clientname] [ -verbose | -debug] = check communication between client and server; to find communication problem; can be executed either from Master or Client

# vmquery -m [-ev in version 3.4] mediaID = tape density, slot number, volume pool assigned, tape location [on/offsite], vault session ID for tape location if offsite; please take note of the result robot type, if it is set to TLD-Tape Library, the tape is in silo, otherwise, it'll show NONE- Not Robotic which is the opposite

# vmquery -h hostname -b -pn Scratch | grep -i TLD | grep -i HCART [| 3] | grep -i E | wc -l = this is for checking of scratch tapes where the hostname is the server where the robot resides or being controlled; Scratch refers to the pool name

# bpgetmedia -p Scratch [| wc -l]

EH0466 8 800 20

EH0901 8 800 20

...

# echo "s d" | tldtest -r /dev/rac/c10t6d0 | grep mediaID = checks for media availability in the robot; -r here refers to the device file/path of the robot [at least for HP-UX]; this shows the drive ["s d"] or ["s s"] for slot

# tpclean -C drivenumber = clean a drive

# vmcheckxxx -rt tld -rn robotnumber = lists the tapes [TLD] currently in the robot; will use current host if neither -h hostname nor -rh robothost is specified

# /usr/openv/netbackup/bin/admincmd/bpdbjobs -report | grep clientname | grep policyname | grep schedule = generates a report of jobs that are done - successful or not, queued, and/or active

# bperror -jobid jobnumber_from_bpdbjobs_o/p -U = provides details about a particular job that ran; usually includes the files/directories being backed up, media used, and error - if failed

# bperror -L -backstat -columns -hours_ago HH | awk '{if (/CLIENT/) ORS="\n" ; else ORS=" "} {print $0}' | grep -i clientname | awk '{print $1, $2, $4, $5, $6, $12, $14, $16, $17, $18, $19}' | grep -i policyname = this is an alternative way of getting the error code from logs

# bpdbjobs [-restart | -cancel | -cancel_all] jobID_number

# telnet clientname bpcd = to check if bpcd daemon is accepting connection; usually executed from Master

# netstat -an | grep -i bpcd = should return bpcd with LISTEN status; see previous blog [and click the Google ads as well, he he]

# bpclntcmd [-ip IP_of_Master | -hn hostname_of_Master] -pn = hostname resolution; resolve multiple host interfaces on clients

# bpplclients | grep -i clientname = lists the client - if present - that is assigned to the Master

# bppllist | grep -i policy_pattern = shows the list of policies accdg. to pattern

# bppllist -byclient clientname -U = lists the policy defined or being used by a particular client

# bppllist policy_name -L -U = a detailed display of properties of a particular policy

# bpdbm -ctime 1161224987 = this will convert it to common format of date and time

1161224987 = Thu Oct 19 10:29:47 2006

# bpimagelist -client clientname -backupid clientname_numberFromANF = I'm not sure if ANF setup is true for all

# bpimagelist -client clientname -d MM/DD/YYYY HH:mm -e MM/DD/YYYY HH:mm -U [-m mediaID] [-L] = this is executed from the Master; checks for saved images from the date specified; note that images saved here are may be limited [not sure 'though depending on the infra setup]

# bpmedialist -U mcontents -m mediaID = list media contents

# bpmedia [-freeze | -unfreeze | -suspend] -ev mediaID

# bpexpdate -d 0 -ev mediaID -force = expire a tape

And lastly, you can run a manual backup as:

# echo "/var/opt" > /home/user01/nbfile

# bpbackup -i -c classname -s schedule -L /path/to/log/file -f /home/user01/nbfile

or

# bpbackup -p policyname -s schedule -L /path/to/log/file /usr/openv/netbackup/bp*

You might be interested on the what tape device files means such as this:

/dev/rmt/cXtYdZBESTnb

where:

BEST = operational capabilities required including the highest density/format and data compression, if supported

n = no rewind

b = Berkeley style; after file closes, tape is not repositioned in anyway

Please note that this was added on 31 December, 2008 @ 17:53:35...

I almost forgot regarding ejecting a tape, which is as crucial as any other task. I'm not sure but I knew I read it somewhere: eject the tape on the robot first before doing it via `nbmenu`. Anyway - sorry, I accept the fact that I do forget things, often -, here is the way to do it:

1. Access the robot via `robtest` [careful with this command, for it can do nasty things to your robot].
2. Before doing anything stupid, please consider the tape if NOT used by other backup jobs.
3. Select from the classification of tapes you want to move [TLD 0, etc.].
4. Execute the following:

"s d" - to check on the contents of the tape drives or identify the drives that has tape (Contains Cartridge =yes, Barcode = XXXXX)
"s s" - check on the content of the library or identify the empty slots (re-inventory)
"m d# s#" - move the tape from drive to slot
"s d" - to verify that the drive is empty after move
"s s" - to verify that the slot has the tape

5. Quit.
6. Now, you can go to the NBU level - which is pretty straightforward - to eject the tape via textual user interface [nbmenu] or CLI:

vmchange -h Master -multi_eject -res -ml list_of_tapes_delimited_by_colon -rt robot_type -rn robot_number -rh MM -sc -verbose

So I guess, this is it. Hope it will help. If you find anything wrong with these tutorials or guides [means, included are the previous], please do leave a message, and I'd be more than happy to check and learn from it. Parking..

Filesystem extention - fsadm errno2

Hi, This will be my first contribution for my friend's blog site. Well, This is just another urgent file system increase for other team restoration task. Basically what happen was after I extended the VG and about to extend the LV an error occurred indicating "vxfs fsadm: cannot open /oracle/A6C/mirrlogA/lost+found/.fsadm - errno 2".

The issue was resolved by recreating the lost+found then re-executing the fsadm.

[root@dagz:/oracle/A6C/origlogB]
# lvextend -L 20000 /dev/vg_A6C_00/lv_mirrlogA
Logical volume "/dev/vg_A6C_00/lv_mirrlogA" has been successfully extended.
Volume Group configuration for /dev/vg_A6C_00 has been saved in /etc/lvmconf/vg_A6C_00.conf

[root@dagz:/oracle/A6C/origlogB]
# fsadm -b 20000m /oracle/A6C/mirrlogA
fsadm: /etc/default/fs is used for determining the file system type
vxfs fsadm: cannot open /oracle/A6C/mirrlogA/lost+found/.fsadm - errno 2

[root@dagz:/oracle/A6C/origlogB]
# cd /oracle/A6C/mirrlogA

[root@dagz:/oracle/A6C/mirrlogA]
# mklost+found
creating slots...
removing dummy files...
done
drwxr-xr-x 2 root sys 4096 Oct 15 10:59 /oracle/A6C/mirrlogA/lost+found

[root@dagz:/oracle/A6C/mirrlogA]
# fsadm -b 20000m /oracle/A6C/mirrlogA
fsadm: /etc/default/fs is used for determining the file system type
fsadm: /dev/vg_A6C_00/rlv_mirrlogA is currently 5120000 sectors - size will be increased

[root@dagz:/oracle/A6C/mirrlogA]
# bdf /oracle/A6C/mirrlogA
Filesystem kbytes used avail %used Mounted on
/dev/vg_A6C_00/lv_mirrlogA
20480000 4102261 15354200 21% /oracle/A6C/mirrlogA

Microsoft’s take on virtualization: re-print edition

By Anna Valmero
INQUIRER.net
First Posted 18:51:00 09/26/2008

MANILA, Philippines -- “The IT industry is in the middle of the next paradigm shift toward virtualization,” said Bill Hilf, general manager of Windows Server and Platform Strategy for Microsoft Corp. Hilf visited the Philippines Friday to promote the release of Microsoft’s virtualization software in the country.

Virtualization, as defined by Microsoft, means having the power to run multiple systems, such as server, storage device, network or even an operating system and applications in just one machine, making a business run its system more efficiently. This virtualization trend comes at the right time to harness the processing power of multi-core processors being deployed in the industry.

As the next trend to shape the IT industry, virtualization can revolutionize the use of computing machines, said Hilf.

A virtualized box can run multiple operating systems compared to a traditional server without virtualization that can handle only one operating system. By deploying virtual machines, one box can serve the function achieved previously by two or more boxes without virtualization. This is also true for desktops.

“Virtualization leads to a lot of savings,” said Hilf.

First, the savings can come from lesser space to be occupied by boxes with virtualization. By having lesser boxes, this also reduces the costs required to cool the server systems. Cooling is required to remove the heat from the server environment, which when reduced translates to lower power utilities. A company can also cut the number of servers it needs to acquire for its business since one box running virtualization can serve the function of two or more servers. Virtualization software packs also offer bulk licensing, as in the case of Microsoft’s premium software products, thus bringing down licensing costs.

Another advantage of virtualization is the ability of a single machine to handle previous-generation applications while running modern applications. This is specifically beneficial to businesses that maintain servers to run old legacy applications.

“By being able to run these old legacy applications on a new, better-performance server, it will help save a lot of money,” Hilf said. Organizations thus achieve better management efficiencies in its resources.

Aside from being able to run multiple operating systems on one desktop, how can the common business person benefit from virtualization?

Explains Hilf: “Imagine this: You have a small device, maybe a laptop or just a small box. Through virtualization, you are able to access via streaming your required applications from a remote server. Technology such as Microsoft's Remote Display Protocol streams keystrokes and mouse clicks to the server and a view of the screen back to the user, who can be located hundreds of miles away from the server or desktop.”

Simply, it can allow users to access a remote PC when he is offsite.


Link: Microsoft’s take on virtualization.

***
I was surprised when I saw and read this article. What the *&%$?!? This is already an "old" [relatively] thing. You should check and get VMWare instead. Well, what's the connection with HP-UX in particular? This can be done on the hardware level: node partitioning [npar]; 'though there is such as virtual partitioning [vpar].

Fibre channel host port is OFFLINE

This may not be your regular error but this is quite useful especially for those who have SAN implementation in their infra [SC21948063]. When we did receive a call from the customer[?], we were compelled to investigate on the raised issue. Basically, I myself, ain't familiar with this. But since this calls for investigation, I accepted the ticket without second thought. Ahhh, ticket! Well I checked first if the box went on a maintenance. So, I did: `uptime`, `date`, `who -r`. Then I came to check for the syslog [HP-UX]; and HW scan [`ioscan -fnCfc`]. Well [why do I use this word too often?!? Well, I don't know either!], there are interesting entries that I found.

[root@server001:/home/user001]
# uptime
11:22am up 323 days, 15:45, 3 users, load average: 0.51, 0.37, 0.31

[root@server001:/home/user001]
# date
Mon Aug 18 11:22:05 EDT 2008

[root@server001:/home/user001]
# who -r
. run-level 4 Sep 29 19:37 4 0 S

[root@server001:/root]
# cat /home/user001/fc_20080818.log
Info for /dev/fcd0

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x454a00
Previous N_Port_id is = 0x454a00
N_Port Node World Wide Name = 0x50060b00003966a5
N_Port Port World Wide Name = 0x50060b00003966a4
Switch Port World Wide Name = 0x204a006069e2147e
Switch Node World Wide Name = 0x1000006069e2147e
Driver state = ONLINE
Hardware Path is = 0/2/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd1

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a7
N_Port Port World Wide Name = 0x50060b00003966a6
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/2/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd2

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x464a00
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a9
N_Port Port World Wide Name = 0x50060b00003966a8
Switch Port World Wide Name = 0x204a006069e2145e
Switch Node World Wide Name = 0x1000006069e2145e
Driver state = ONLINE
Hardware Path is = 0/4/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd3

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966ab
N_Port Port World Wide Name = 0x50060b00003966aa
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/4/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd4

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396885
N_Port Port World Wide Name = 0x50060b0000396884
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/5/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd5

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x686900
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396887
N_Port Port World Wide Name = 0x50060b0000396886
Switch Port World Wide Name = 0x2069006069e207b2
Switch Node World Wide Name = 0x1000006069e207b2
Driver state = ONLINE
Hardware Path is = 0/5/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14


[root@server001:/home/user001]
# cat syslog.out
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan

[root@server001:/home/user001]
#

If you can see from the logs the line that contains this: Driver state = AWAITING_LINK_UP. As per suggestion [which resolved the issue btw] from http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1219074300410+28353475&threadId=1165601, this could be a port issue:

The "AWAITING_LINK_UP" state describes the problem. Check with your SAN team about the status of the switch port (F-port) to which the HBA is connected.

To which the caller affirm that the FC was improperly connected. But still, there could be some other reason for this. I'll wait for the response/action...

Also, if you can see from the syslog, you can run this for more info:

[root@server001:/home/user001]
# /opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a

Reconfiguring an HP-UX (11iv1) kernel

It's whole new experience [heaven]. I thought it will remain a wish. Performing this is a long-shot in our working environment. But, what could be sweeter than a wish coming true? Nada! Of course, other than.. nah, nevermind! Ok, we have a project to have a kernel parameter changed. Upon doing a prework, we have this initial value:

[root@hpux09:/stand/build]
# kmtune -q vx_maxlink
Parameter Current Dyn Planned Module Version
==========================================
vx_maxlink 32767 - 32767

[root@hpux05:/stand/build]

We usually perform these kind of changes via SAM. But, wait! What the f*%$? Where's the vx_maxlink parameter?!? "Uhmm, hey [referring to my colleague], can you check it with L3?" So that's it. There are some parameters that does not show [or linked] on SAM. Changes to be made are to be performed via CLI. So, here are the steps I followed:

[root@hpux09:/stand/build]
# /usr/lbin/sysadm/system_prep -s system

[root@hpux09:/stand/build]
# kmtune -q vx_maxlink
Parameter Current Dyn Planned Module Version
==========================================
vx_maxlink 32767 - 32767

[root@hpux09:/stand/build]
# kmtune -s vx_maxlink=65534

[root@hpux09:/stand/build]
# kmtune -q vx_maxlink
Parameter Current Dyn Planned Module Version
==========================================
vx_maxlink 32767 - 65534

[root@hpux09:/stand/build]
# which mk_kernel
/usr/sbin/mk_kernel

[root@hpux09:/stand/build]
# mk_kernel -s system
Generating module: krm...
Generating module: SEOS...
Compiling conf.c...
Loading the kernel...
Generating kernel symbol table...

[root@hpux09:/stand/build]
# kmupdate

Kernel update request is scheduled.

Default kernel /stand/vmunix will be updated by
newly built kernel /stand/build/vmunix_test
at next system shutdown or startup time.


[root@hpux09:/stand/build]
# shutdown -ry 0
Shutdown cannot be run from a mounted file system -- exiting shutdown.
Change directories to the root volume ("/" will work) and try again.

[root@hpux09:/stand/build]
# cd /

[root@hpux09:/]
# shutdown -ry 0

SHUTDOWN PROGRAM
05/24/08 23:40:31 EDT

Broadcast Message from root (pts/5) Sat May 24 23:40:31...
PLEASE LOG OFF NOW ! ! !
System maintenance about to begin.
All processes will be terminated in 0 seconds.

Broadcast Message from root (pts/5) Sat May 24 23:40:31...
SYSTEM BEING BROUGHT DOWN NOW ! ! !

/sbin/auto_parms: DHCP access is disabled (see /etc/auto_parms.log)

For now, we have to wait for box to come up and check if the change we applied took effect. [Cross-finger] Hoping it did.?! It's driving me crazy [and very excited!].

What the f*%$ have I done?? I am doomed! The kernel parameter didn't change at all. The value is still under planned. My heart raced and pounded. Hey! I'm no superman. Looking for a reason to have the window time extended. Deym! [Temporary, still to hear a LOT about this during our weekly meeting] Fortunate for me (?), the change was so important that it left no choice for the application team, requestor, and box owner to extend the time and allow me to give it another go. But, this time? I got L3's attention! I consulted them, and they gave me an SOP [btw, for the record, of which I'm not aware of and was not provided]. [Another] But, to make sure, I let the L3 do it [I got my hands tied already, so I'm not taking any chances - not now but, definitely will love to do it again, anytime, anywhere!], and check how he did it a bit later. A few, very long, minutes later, he's working his magic. And here's how:

cd /stand/build
ll system [optional but essential]
kmtune -q vx_maxlink
/usr/lbin/sysadm/system_prep -s system
kmtune -s vx_maxlink=65534 -S ./system # This is what I missed; writing to system file
/usr/sbin/mk_kernel -s system
kmupdate
cd ..
cp -p system system_prev
mv build/system .
kmtune -q vx_maxlink
[now all I need is to reboot the box, and done!]

Well, folks, I hope you learned new. For me? I learned a TON!

And oh, make sure to watch out for the /stand FS getting full. You might end up just like it. I tell you, it's nasty. May be giving a system_prep will clear it... Well, just a thought. Good luck to us all.


This was added a bit later [July something of 2008].

Here's the procedure for HPUX 11.23: (explanation? Later)

# mv /stand/system /stand/system.orig

# kconfig -e /stand/system

CEdit /stand/system file and remove all "Tunable Parameters".
Copy and paste Tunable parameters ( lined between START KERNEL PARM and END KERNEL PARM) from /tmp/logfile.
Save and exit.

# kconfig -i /stand/system

# shutdown -r 0