This may not be your regular error but this is quite useful especially for those who have SAN implementation in their infra [SC21948063]. When we did receive a call from the customer[?], we were compelled to investigate on the raised issue. Basically, I myself, ain't familiar with this. But since this calls for investigation, I accepted the ticket without second thought. Ahhh, ticket! Well I checked first if the box went on a maintenance. So, I did: `uptime`, `date`, `who -r`. Then I came to check for the syslog [HP-UX]; and HW scan [`ioscan -fnCfc`]. Well [why do I use this word too often?!? Well, I don't know either!], there are interesting entries that I found.
[root@server001:/home/user001]
# uptime
11:22am up 323 days, 15:45, 3 users, load average: 0.51, 0.37, 0.31
[root@server001:/home/user001]
# date
Mon Aug 18 11:22:05 EDT 2008
[root@server001:/home/user001]
# who -r
. run-level 4 Sep 29 19:37 4 0 S
[root@server001:/root]
# cat /home/user001/fc_20080818.log
Info for /dev/fcd0
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x454a00
Previous N_Port_id is = 0x454a00
N_Port Node World Wide Name = 0x50060b00003966a5
N_Port Port World Wide Name = 0x50060b00003966a4
Switch Port World Wide Name = 0x204a006069e2147e
Switch Node World Wide Name = 0x1000006069e2147e
Driver state = ONLINE
Hardware Path is = 0/2/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01
/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14
Info for /dev/fcd1
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a7
N_Port Port World Wide Name = 0x50060b00003966a6
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/2/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01
/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14
Info for /dev/fcd2
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x464a00
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a9
N_Port Port World Wide Name = 0x50060b00003966a8
Switch Port World Wide Name = 0x204a006069e2145e
Switch Node World Wide Name = 0x1000006069e2145e
Driver state = ONLINE
Hardware Path is = 0/4/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01
/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14
Info for /dev/fcd3
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966ab
N_Port Port World Wide Name = 0x50060b00003966aa
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/4/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01
/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14
Info for /dev/fcd4
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396885
N_Port Port World Wide Name = 0x50060b0000396884
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/5/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01
/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14
Info for /dev/fcd5
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x686900
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396887
N_Port Port World Wide Name = 0x50060b0000396886
Switch Port World Wide Name = 0x2069006069e207b2
Switch Node World Wide Name = 0x1000006069e207b2
Driver state = ONLINE
Hardware Path is = 0/5/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01
/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14
[root@server001:/home/user001]
# cat syslog.out
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:
"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:
"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:
"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:
"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:
"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:
"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
[root@server001:/home/user001]
#
If you can see from the logs the line that contains this: Driver state = AWAITING_LINK_UP. As per suggestion [which resolved the issue btw] from http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1219074300410+28353475&threadId=1165601, this could be a port issue:
The "AWAITING_LINK_UP" state describes the problem. Check with your SAN team about the status of the switch port (F-port) to which the HBA is connected.
To which the caller affirm that the FC was improperly connected. But still, there could be some other reason for this. I'll wait for the response/action...
Also, if you can see from the syslog, you can run this for more info:
[root@server001:/home/user001]
# /opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Showing posts with label ITRC. Show all posts
Showing posts with label ITRC. Show all posts
Monday, August 18, 2008
Friday, April 11, 2008
Ignite backup error - The list_expander command failed
Hello team, I'd like to share one of the common errors you will find during ignite backup.
[root@hpux02:/root]
# /opt/ignite/bin/make_net_recovery -s hpux03 -x inc_entire=vg00
* Creating NFS mount directories for configuration files.
======= 04/11/08 10:56:30 SST Started /opt/ignite/bin/make_net_recovery. (Fri
Apr 11 10:56:30 SST 2008)
@(#) Ignite-UX Revision B.5.4.50
@(#) net_recovery (opt) $Revision: 10.637 $
* Testing pax for needed patch
* Passed pax tests.
* Checking Versions of Recovery Tools
ERROR: Cannot stat device file: /dev/old_vg_apps/lv_mom: No such file or
directory (errno = 2). Check /etc/fstab for a bad entry.
ERROR: The list_expander command failed. This could be due to a problem with
the -x options specified - see messages above.
======= 04/11/08 10:57:11 SST make_net_recovery completed unsuccessfully
[root@hpux02:/root]
#
AND/OR
# /opt/ignite/lbin/list_expander
ERROR: Cannot stat device file: /dev/old_vg_hr03/lv_u01: No such file or
directory (errno = 2). Check /etc/fstab for a bad entry.
[root@hpux02:/root]
#
This occurs when certain FS is no longer in use and was not removed in /etc/fstab. I check it using:
# bdf <block_device_name>
# bdf /dev/old_vg_apps/lv_mom
OR
# mount -v grep -i <pattern>
AND/OR
# lvdisplay <block_device_name>
# lvdisplay /dev/old_vg_apps/lv_mom
If it does NOT return any of your expected result, then it is safe to say that the FS is NOT is use. From here, we can proceed by commenting [deleting] this but, make sure to have a backup just in case. Systems admins know this by heart. "He who laughs last, made a backup!". Proceeding..
# cp -p /etc/fstab /etc/fstab.backup1.20080411
# vi /etc/fstab
....
/dev/old_vg_hr03/lv_u01 /old/var/opt/vghr03/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr04/lv_u01 /old/var/opt/vghr04/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr05/lv_u01 /old/var/opt/vghr05/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr06/lv_u01 /old/var/opt/vghr06/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr07/lv_u01 /old/var/opt/vghr07/u01 vxfs delaylog,mincache=direct 0 2
....
So, our target here is to comment these lines so that it will not be read by ignite. For this case, we can comment it one by one. But for cases where we want to comment hundreds of lines, it'll take a lot of time. So I searched for some vi commands that will do this task for me. And I found it here: http://sparky.rice.edu/~hartigan/vi.html.
I executed this in command mode: :218,$s/^/\#
This means that, I'd like to insert a "#" character at the beginning of each line from line 218 to the last. Save and exit. Then re-ran ignite. As of this writing, it's running smoothly.
Interesting topic: http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1207883792669+28353475&threadId=996179
[root@hpux02:/root]
# /opt/ignite/bin/make_net_recovery -s hpux03 -x inc_entire=vg00
* Creating NFS mount directories for configuration files.
======= 04/11/08 10:56:30 SST Started /opt/ignite/bin/make_net_recovery. (Fri
Apr 11 10:56:30 SST 2008)
@(#) Ignite-UX Revision B.5.4.50
@(#) net_recovery (opt) $Revision: 10.637 $
* Testing pax for needed patch
* Passed pax tests.
* Checking Versions of Recovery Tools
ERROR: Cannot stat device file: /dev/old_vg_apps/lv_mom: No such file or
directory (errno = 2). Check /etc/fstab for a bad entry.
ERROR: The list_expander command failed. This could be due to a problem with
the -x options specified - see messages above.
======= 04/11/08 10:57:11 SST make_net_recovery completed unsuccessfully
[root@hpux02:/root]
#
AND/OR
# /opt/ignite/lbin/list_expander
ERROR: Cannot stat device file: /dev/old_vg_hr03/lv_u01: No such file or
directory (errno = 2). Check /etc/fstab for a bad entry.
[root@hpux02:/root]
#
This occurs when certain FS is no longer in use and was not removed in /etc/fstab. I check it using:
# bdf <block_device_name>
# bdf /dev/old_vg_apps/lv_mom
OR
# mount -v grep -i <pattern>
AND/OR
# lvdisplay <block_device_name>
# lvdisplay /dev/old_vg_apps/lv_mom
If it does NOT return any of your expected result, then it is safe to say that the FS is NOT is use. From here, we can proceed by commenting [deleting] this but, make sure to have a backup just in case. Systems admins know this by heart. "He who laughs last, made a backup!". Proceeding..
# cp -p /etc/fstab /etc/fstab.backup1.20080411
# vi /etc/fstab
....
/dev/old_vg_hr03/lv_u01 /old/var/opt/vghr03/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr04/lv_u01 /old/var/opt/vghr04/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr05/lv_u01 /old/var/opt/vghr05/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr06/lv_u01 /old/var/opt/vghr06/u01 vxfs delaylog,mincache=direct 0 2
/dev/old_vg_hr07/lv_u01 /old/var/opt/vghr07/u01 vxfs delaylog,mincache=direct 0 2
....
So, our target here is to comment these lines so that it will not be read by ignite. For this case, we can comment it one by one. But for cases where we want to comment hundreds of lines, it'll take a lot of time. So I searched for some vi commands that will do this task for me. And I found it here: http://sparky.rice.edu/~hartigan/vi.html.
I executed this in command mode: :218,$s/^/\#
This means that, I'd like to insert a "#" character at the beginning of each line from line 218 to the last. Save and exit. Then re-ran ignite. As of this writing, it's running smoothly.
Interesting topic: http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1207883792669+28353475&threadId=996179
Thursday, April 10, 2008
Ignite backup
I was surprised to receive a PM from my colleague. As part of the process we follow, we need to make an ignite backup of system we do patching on. Most of the time, it goes smooth but they can never be perfect, 82-0 is just impossible. Anyway, I have encountered this issue a couple of times already. I've read some discussions on this but still trying to understand. Way to far, to many to learn! Going back, here's the error:
...
* Saving the information about archive to
/var/opt/ignite/recovery/previews
* Creating The Networking Archive
Permission denied
ERROR: Unable to mount or write
server1:/var/opt/ignite/recovery/archives/client_svr1
On server1 you may need to:
mkdir -p /var/opt/ignite/recovery/archives/client_svr1
chown bin:bin /var/opt/ignite/recovery/archives/client_svr1
vi /etc/exports (add or modify an entry). The /etc/exports file on
"server1" should contain the entry:
"/var/opt/ignite/recovery/archives/client_svr1 -anon=2,access=client_svr1".
After editing the /etc/exports file, run exportfs -av
If you need to change the owner of the directory,
you will also need to re-export the directory.
See make_net_recovery(1M) for more information.
ERROR: Failed to Create NFS mount Archive directory.
======= 04/10/08 09:07:54 SST make_net_recovery completed unsuccessfully
[root@server1:/var/opt/ignite/recovery]
#
I followed the suggestion here: `mkdir`; `chmod`; and `exportfs` but to no avail. Well, the workaround is NOT to limit access to the specific client, i.e., delete the entry "access=client_svr1" from /etc/exports; then run re-export the FSes. Now, it's working well. As mentioned earlier, it's just a workaround. So we need to restore the entries in the config files.
Read on: http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1017493&admit=109447627+1207790553782+28353475
...
* Saving the information about archive to
/var/opt/ignite/recovery/previews
* Creating The Networking Archive
Permission denied
ERROR: Unable to mount or write
server1:/var/opt/ignite/recovery/archives/client_svr1
On server1 you may need to:
mkdir -p /var/opt/ignite/recovery/archives/client_svr1
chown bin:bin /var/opt/ignite/recovery/archives/client_svr1
vi /etc/exports (add or modify an entry). The /etc/exports file on
"server1" should contain the entry:
"/var/opt/ignite/recovery/archives/client_svr1 -anon=2,access=client_svr1".
After editing the /etc/exports file, run exportfs -av
If you need to change the owner of the directory,
you will also need to re-export the directory.
See make_net_recovery(1M) for more information.
ERROR: Failed to Create NFS mount Archive directory.
======= 04/10/08 09:07:54 SST make_net_recovery completed unsuccessfully
[root@server1:/var/opt/ignite/recovery]
#
I followed the suggestion here: `mkdir`; `chmod`; and `exportfs` but to no avail. Well, the workaround is NOT to limit access to the specific client, i.e., delete the entry "access=client_svr1" from /etc/exports; then run re-export the FSes. Now, it's working well. As mentioned earlier, it's just a workaround. So we need to restore the entries in the config files.
Read on: http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1017493&admit=109447627+1207790553782+28353475
Subscribe to:
Posts (Atom)