Fibre channel host port is OFFLINE

This may not be your regular error but this is quite useful especially for those who have SAN implementation in their infra [SC21948063]. When we did receive a call from the customer[?], we were compelled to investigate on the raised issue. Basically, I myself, ain't familiar with this. But since this calls for investigation, I accepted the ticket without second thought. Ahhh, ticket! Well I checked first if the box went on a maintenance. So, I did: `uptime`, `date`, `who -r`. Then I came to check for the syslog [HP-UX]; and HW scan [`ioscan -fnCfc`]. Well [why do I use this word too often?!? Well, I don't know either!], there are interesting entries that I found.

[root@server001:/home/user001]
# uptime
11:22am up 323 days, 15:45, 3 users, load average: 0.51, 0.37, 0.31

[root@server001:/home/user001]
# date
Mon Aug 18 11:22:05 EDT 2008

[root@server001:/home/user001]
# who -r
. run-level 4 Sep 29 19:37 4 0 S

[root@server001:/root]
# cat /home/user001/fc_20080818.log
Info for /dev/fcd0

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x454a00
Previous N_Port_id is = 0x454a00
N_Port Node World Wide Name = 0x50060b00003966a5
N_Port Port World Wide Name = 0x50060b00003966a4
Switch Port World Wide Name = 0x204a006069e2147e
Switch Node World Wide Name = 0x1000006069e2147e
Driver state = ONLINE
Hardware Path is = 0/2/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd1

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a7
N_Port Port World Wide Name = 0x50060b00003966a6
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/2/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd2

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x464a00
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966a9
N_Port Port World Wide Name = 0x50060b00003966a8
Switch Port World Wide Name = 0x204a006069e2145e
Switch Node World Wide Name = 0x1000006069e2145e
Driver state = ONLINE
Hardware Path is = 0/4/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd3

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b00003966ab
N_Port Port World Wide Name = 0x50060b00003966aa
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/4/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd4

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396885
N_Port Port World Wide Name = 0x50060b0000396884
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/5/2/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14

Info for /dev/fcd5

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012ba
PCI Mode = PCI-X 66 MHz
ISP Code version = 3.2.162
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x686900
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000396887
N_Port Port World Wide Name = 0x50060b0000396886
Switch Port World Wide Name = 0x2069006069e207b2
Switch Node World Wide Name = 0x1000006069e207b2
Driver state = ONLINE
Hardware Path is = 0/5/2/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.11.01

/ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Jul 16 2003,18:50:14


[root@server001:/home/user001]
# cat syslog.out
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:02 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:06 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_2_1_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208071 -r /adapters/events/ql_adapter/0_2_1_1 -n 293208079 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_4_2_1" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208087 -r /adapters/events/ql_adapter/0_4_2_1 -n 293208080 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:07 server001 EMS [4474]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource:

"/adapters/events/ql_adapter/0_5_2_0" (Threshold: >= " 3") Execute the following command to obtain event details:

/opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:10 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 14:00:14 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 17 18:00:20 server001 vmunix: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/4/2/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/5/2/0: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan
Aug 18 11:20:50 server001 vmunix: 0/2/1/1: Fibre channel host port is OFFLINE, can not scan

[root@server001:/home/user001]
#

If you can see from the logs the line that contains this: Driver state = AWAITING_LINK_UP. As per suggestion [which resolved the issue btw] from http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1219074300410+28353475&threadId=1165601, this could be a port issue:

The "AWAITING_LINK_UP" state describes the problem. Check with your SAN team about the status of the switch port (F-port) to which the HBA is connected.

To which the caller affirm that the FC was improperly connected. But still, there could be some other reason for this. I'll wait for the response/action...

Also, if you can see from the syslog, you can run this for more info:

[root@server001:/home/user001]
# /opt/resmon/bin/resdata -R 293208092 -r /adapters/events/ql_adapter/0_5_2_0 -n 293208081 -a