Problem
On a VIOS we see a alot of duplicate paths for most of the LUNs:
VIOS:> lspath
status name parent connection
Enabled hdisk1 fscsi0 5005076801104c32,0
Enabled hdisk2 fscsi0 5005076801104c32,1000000000000
Enabled hdisk3 fscsi0 5005076801104c32,2000000000000
Enabled hdisk4 fscsi0 5005076801104c32,3000000000000
Enabled hdisk5 fscsi0 5005076801104c32,4000000000000
Enabled hdisk6 fscsi0 5005076801104c32,5000000000000
Enabled hdisk7 fscsi0 5005076801104c32,6000000000000
Enabled hdisk8 fscsi0 5005076801104c32,7000000000000
Enabled hdisk9 fscsi0 5005076801104c32,8000000000000
Enabled hdisk1 fscsi0 5005076801104a2a,0
Enabled hdisk2 fscsi0 5005076801104a2a,1000000000000
Enabled hdisk3 fscsi0 5005076801104a2a,2000000000000
Enabled hdisk4 fscsi0 5005076801104a2a,3000000000000
Enabled hdisk5 fscsi0 5005076801104a2a,4000000000000
Enabled hdisk6 fscsi0 5005076801104a2a,5000000000000
Enabled hdisk7 fscsi0 5005076801104a2a,6000000000000
Enabled hdisk8 fscsi0 5005076801104a2a,7000000000000
Enabled hdisk9 fscsi0 5005076801104a2a,8000000000000
Enabled hdisk0 fscsi0 500507680140dc81,0
Enabled hdisk0 fscsi0 500507680140dc91,0
Enabled hdisk1 fscsi1 5005076801304a2a,0
Enabled hdisk2 fscsi1 5005076801304a2a,1000000000000
Enabled hdisk3 fscsi1 5005076801304a2a,2000000000000
Enabled hdisk4 fscsi1 5005076801304a2a,3000000000000
Enabled hdisk5 fscsi1 5005076801304a2a,4000000000000
Enabled hdisk6 fscsi1 5005076801304a2a,5000000000000
Enabled hdisk7 fscsi1 5005076801304a2a,6000000000000
Enabled hdisk8 fscsi1 5005076801304a2a,7000000000000
Enabled hdisk9 fscsi1 5005076801304a2a,8000000000000
Enabled hdisk1 fscsi1 5005076801304c32,0
Enabled hdisk2 fscsi1 5005076801304c32,1000000000000
Enabled hdisk3 fscsi1 5005076801304c32,2000000000000
Enabled hdisk4 fscsi1 5005076801304c32,3000000000000
Enabled hdisk5 fscsi1 5005076801304c32,4000000000000
Enabled hdisk6 fscsi1 5005076801304c32,5000000000000
Enabled hdisk7 fscsi1 5005076801304c32,6000000000000
Enabled hdisk8 fscsi1 5005076801304c32,7000000000000
Enabled hdisk9 fscsi1 5005076801304c32,8000000000000
Enabled hdisk0 fscsi1 500507680120dc81,0
Enabled hdisk0 fscsi1 500507680120dc91,0
Enabled hdisk1 fscsi0 5005076801204c32,0
Enabled hdisk2 fscsi0 5005076801204c32,1000000000000
Enabled hdisk3 fscsi0 5005076801204c32,2000000000000
Enabled hdisk4 fscsi0 5005076801204c32,3000000000000
Enabled hdisk5 fscsi0 5005076801204c32,4000000000000
Enabled hdisk6 fscsi0 5005076801204c32,5000000000000
Enabled hdisk7 fscsi0 5005076801204c32,6000000000000
Enabled hdisk8 fscsi0 5005076801204c32,7000000000000
Enabled hdisk9 fscsi0 5005076801204c32,8000000000000
Enabled hdisk1 fscsi0 5005076801204a2a,0
Enabled hdisk2 fscsi0 5005076801204a2a,1000000000000
Enabled hdisk3 fscsi0 5005076801204a2a,2000000000000
Enabled hdisk4 fscsi0 5005076801204a2a,3000000000000
Enabled hdisk5 fscsi0 5005076801204a2a,4000000000000
Enabled hdisk6 fscsi0 5005076801204a2a,5000000000000
Enabled hdisk7 fscsi0 5005076801204a2a,6000000000000
Enabled hdisk8 fscsi0 5005076801204a2a,7000000000000
Enabled hdisk9 fscsi0 5005076801204a2a,8000000000000
Enabled hdisk10 fscsi0 5005076801104c32,9000000000000
Enabled hdisk11 fscsi0 5005076801104c32,a000000000000
Enabled hdisk12 fscsi0 5005076801104c32,b000000000000
Enabled hdisk10 fscsi0 5005076801204c32,9000000000000
Enabled hdisk11 fscsi0 5005076801204c32,a000000000000
Enabled hdisk12 fscsi0 5005076801204c32,b000000000000
Enabled hdisk10 fscsi0 5005076801104a2a,9000000000000
Enabled hdisk11 fscsi0 5005076801104a2a,a000000000000
Enabled hdisk12 fscsi0 5005076801104a2a,b000000000000
Enabled hdisk10 fscsi0 5005076801204a2a,9000000000000
Enabled hdisk11 fscsi0 5005076801204a2a,a000000000000
Enabled hdisk12 fscsi0 5005076801204a2a,b000000000000
Enabled hdisk10 fscsi1 5005076801304a2a,9000000000000
Enabled hdisk11 fscsi1 5005076801304a2a,a000000000000
Enabled hdisk12 fscsi1 5005076801304a2a,b000000000000
Enabled hdisk10 fscsi1 5005076801304c32,9000000000000
Enabled hdisk11 fscsi1 5005076801304c32,a000000000000
Enabled hdisk12 fscsi1 5005076801304c32,b000000000000
Enabled sasdisk0 sas0 20000,0
Enabled hdisk13 fscsi0 5005076801104c32,c000000000000
Enabled hdisk14 fscsi0 5005076801104c32,d000000000000
Enabled hdisk15 fscsi0 5005076801104c32,e000000000000
Enabled hdisk16 fscsi0 5005076801104c32,f000000000000
Enabled hdisk17 fscsi0 5005076801104c32,10000000000000
Enabled hdisk13 fscsi0 5005076801204c32,c000000000000
Enabled hdisk14 fscsi0 5005076801204c32,d000000000000
Enabled hdisk15 fscsi0 5005076801204c32,e000000000000
Enabled hdisk16 fscsi0 5005076801204c32,f000000000000
Enabled hdisk17 fscsi0 5005076801204c32,10000000000000
Enabled hdisk13 fscsi0 5005076801104a2a,c000000000000
Enabled hdisk14 fscsi0 5005076801104a2a,d000000000000
Enabled hdisk15 fscsi0 5005076801104a2a,e000000000000
Enabled hdisk16 fscsi0 5005076801104a2a,f000000000000
Enabled hdisk17 fscsi0 5005076801104a2a,10000000000000
Enabled hdisk13 fscsi0 5005076801204a2a,c000000000000
Enabled hdisk14 fscsi0 5005076801204a2a,d000000000000
Enabled hdisk15 fscsi0 5005076801204a2a,e000000000000
Enabled hdisk16 fscsi0 5005076801204a2a,f000000000000
Enabled hdisk17 fscsi0 5005076801204a2a,10000000000000
Enabled hdisk13 fscsi1 5005076801304a2a,c000000000000
Enabled hdisk14 fscsi1 5005076801304a2a,d000000000000
Enabled hdisk15 fscsi1 5005076801304a2a,e000000000000
Enabled hdisk16 fscsi1 5005076801304a2a,f000000000000
Enabled hdisk17 fscsi1 5005076801304a2a,10000000000000
Enabled hdisk13 fscsi1 5005076801304c32,c000000000000
Enabled hdisk14 fscsi1 5005076801304c32,d000000000000
Enabled hdisk15 fscsi1 5005076801304c32,e000000000000
Enabled hdisk16 fscsi1 5005076801304c32,f000000000000
Enabled hdisk17 fscsi1 5005076801304c32,10000000000000
Enabled hdisk18 fscsi0 500507680140dc81,1000000000000
Enabled hdisk18 fscsi0 500507680140dc91,1000000000000
Enabled hdisk18 fscsi1 500507680120dc81,1000000000000
Enabled hdisk18 fscsi1 500507680120dc91,1000000000000For example: hdisk17 has 6 paths instead of 4:
VIOS:> lspath -dev hdisk17
status name parent connection
Enabled hdisk17 fscsi0 5005076801104c32,10000000000000
Enabled hdisk17 fscsi0 5005076801204c32,10000000000000
Enabled hdisk17 fscsi0 5005076801104a2a,10000000000000
Enabled hdisk17 fscsi0 5005076801204a2a,10000000000000
Enabled hdisk17 fscsi1 5005076801304a2a,10000000000000
Enabled hdisk17 fscsi1 5005076801304c32,10000000000000But yet a newly added hdisk18 has only 4:
VIOS:> lspath -dev hdisk18
status name parent connection
Enabled hdisk18 fscsi0 500507680140dc81,1000000000000
Enabled hdisk18 fscsi0 500507680140dc91,1000000000000
Enabled hdisk18 fscsi1 500507680120dc81,1000000000000
Enabled hdisk18 fscsi1 500507680120dc91,1000000000000The datapath command shows the additional paths also:
VIOS:> datapath query device
Total Dual Active and Active/Asymmetric Devices : 19
DEV#: 1 DEVICE NAME: hdisk1 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E82619000000000002EF6
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path2 OPEN NORMAL 21 0
1 fscsi1/path3 OPEN NORMAL 1719645 0
2 fscsi0/path0 OPEN NORMAL 1087376 0
3* fscsi0/path1 OPEN NORMAL 21 0
4 fscsi0/path4 OPEN NORMAL 1087492 0
5* fscsi0/path5 OPEN NORMAL 21 0
DEV#: 2 DEVICE NAME: hdisk2 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E82619000000000002FF4
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi1/path2 OPEN NORMAL 1820666 0
1* fscsi1/path3 OPEN NORMAL 21 0
2* fscsi0/path0 OPEN NORMAL 21 0
3 fscsi0/path1 OPEN NORMAL 1156226 0
4* fscsi0/path4 OPEN NORMAL 21 0
5 fscsi0/path5 OPEN NORMAL 1154046 0
DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E82619000000000002FF5
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path2 OPEN NORMAL 35 0
1 fscsi1/path3 OPEN NORMAL 1435944 0
2 fscsi0/path0 OPEN NORMAL 883921 0
3* fscsi0/path1 OPEN NORMAL 28 0
4 fscsi0/path4 OPEN NORMAL 884695 0
5* fscsi0/path5 OPEN NORMAL 35 0
DEV#: 4 DEVICE NAME: hdisk4 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E8261900000000000303E
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path2 OPEN NORMAL 21 0
1 fscsi1/path3 OPEN NORMAL 285190 0
2 fscsi0/path0 OPEN NORMAL 98426 0
3* fscsi0/path1 OPEN NORMAL 21 0
4 fscsi0/path4 OPEN NORMAL 97737 0
5* fscsi0/path5 OPEN NORMAL 21 0
DEV#: 5 DEVICE NAME: hdisk5 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000030EF
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path2 CLOSE NORMAL 70 0
1 fscsi1/path3 CLOSE NORMAL 80 0
2 fscsi0/path0 CLOSE NORMAL 71 0
3* fscsi0/path1 CLOSE NORMAL 70 0
4 fscsi0/path4 CLOSE NORMAL 72 0
5* fscsi0/path5 CLOSE NORMAL 70 0
DEV#: 6 DEVICE NAME: hdisk6 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000030F0
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi1/path2 CLOSE NORMAL 78 0
1* fscsi1/path3 CLOSE NORMAL 70 0
2* fscsi0/path0 CLOSE NORMAL 70 0
3 fscsi0/path1 CLOSE NORMAL 71 0
4* fscsi0/path4 CLOSE NORMAL 70 0
5 fscsi0/path5 CLOSE NORMAL 74 0
DEV#: 7 DEVICE NAME: hdisk7 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E82619000000000003270
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi1/path2 OPEN NORMAL 276311 0
1* fscsi1/path3 OPEN NORMAL 21 0
2* fscsi0/path0 OPEN NORMAL 21 0
3 fscsi0/path1 OPEN NORMAL 97662 0
4* fscsi0/path4 OPEN NORMAL 21 0
5 fscsi0/path5 OPEN NORMAL 97489 0
DEV#: 8 DEVICE NAME: hdisk8 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000036B8
=========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path2 OPEN NORMAL 21 0
1 fscsi1/path3 OPEN NORMAL 303137 0
2 fscsi0/path0 OPEN NORMAL 125066 0
3* fscsi0/path1 OPEN NORMAL 28 0
4 fscsi0/path4 OPEN NORMAL 126426 0
5* fscsi0/path5 OPEN NORMAL 21 0
DEV#: 9 DEVICE NAME: hdisk9 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000037D1
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path2 OPEN NORMAL 21 0
1 fscsi1/path3 OPEN NORMAL 308114 0
2 fscsi0/path0 OPEN NORMAL 124842 0
3* fscsi0/path1 OPEN NORMAL 21 0
4 fscsi0/path4 OPEN NORMAL 124216 0
5* fscsi0/path5 OPEN NORMAL 21 0
DEV#: 0 DEVICE NAME: hdisk0 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018086E40800000000000074
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 OPEN NORMAL 7093852 0
1* fscsi0/path1 OPEN NORMAL 52 0
2 fscsi1/path2 OPEN NORMAL 7385644 0
3* fscsi1/path3 OPEN NORMAL 52 0
DEV#: 10 DEVICE NAME: hdisk10 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000046E4
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path4 OPEN NORMAL 28 0
1 fscsi1/path5 OPEN NORMAL 6991560 0
2 fscsi0/path0 OPEN NORMAL 4964095 0
3 fscsi0/path1 OPEN NORMAL 4963196 0
4* fscsi0/path2 OPEN NORMAL 28 0
5* fscsi0/path3 OPEN NORMAL 28 0
DEV#: 11 DEVICE NAME: hdisk11 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E82619000000000004A00
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path4 OPEN NORMAL 28 0
1 fscsi1/path5 OPEN NORMAL 735702974 0
2 fscsi0/path0 OPEN NORMAL 445782223 0
3 fscsi0/path1 OPEN NORMAL 445781860 0
4* fscsi0/path2 OPEN NORMAL 28 0
5* fscsi0/path3 OPEN NORMAL 28 0
DEV#: 12 DEVICE NAME: hdisk12 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E82619000000000004A02
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi1/path4 OPEN NORMAL 21 0
1 fscsi1/path5 OPEN NORMAL 20965908 0
2 fscsi0/path0 OPEN NORMAL 13563901 0
3 fscsi0/path1 OPEN NORMAL 13564590 0
4* fscsi0/path2 OPEN NORMAL 21 0
5* fscsi0/path3 OPEN NORMAL 21 0
DEV#: 13 DEVICE NAME: hdisk13 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000075CA
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 OPEN NORMAL 893304 0
1 fscsi0/path1 OPEN NORMAL 851758 0
2* fscsi0/path2 OPEN NORMAL 133 0
3* fscsi0/path3 OPEN NORMAL 133 0
4* fscsi1/path4 OPEN NORMAL 133 0
5 fscsi1/path5 OPEN NORMAL 1236751 0
DEV#: 14 DEVICE NAME: hdisk14 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000075CF
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi0/path0 OPEN NORMAL 56 0
1* fscsi0/path1 OPEN NORMAL 56 0
2 fscsi0/path2 OPEN NORMAL 196677 0
3 fscsi0/path3 OPEN NORMAL 155511 0
4 fscsi1/path4 OPEN NORMAL 211308 0
5* fscsi1/path5 OPEN NORMAL 56 0
DEV#: 15 DEVICE NAME: hdisk15 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000075CC
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 OPEN NORMAL 8844974 0
1 fscsi0/path1 OPEN NORMAL 8795463 0
2* fscsi0/path2 OPEN NORMAL 133 0
3* fscsi0/path3 OPEN NORMAL 133 0
4* fscsi1/path4 OPEN NORMAL 133 0
5 fscsi1/path5 OPEN NORMAL 13549327 0
DEV#: 16 DEVICE NAME: hdisk16 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000075CD
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi0/path0 OPEN NORMAL 133 0
1* fscsi0/path1 OPEN NORMAL 133 0
2 fscsi0/path2 OPEN NORMAL 73812 0
3 fscsi0/path3 OPEN NORMAL 31740 0
4 fscsi1/path4 OPEN NORMAL 43271 0
5* fscsi1/path5 OPEN NORMAL 133 0
DEV#: 17 DEVICE NAME: hdisk17 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018E826190000000000075CE
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 OPEN NORMAL 64733 0
1 fscsi0/path1 OPEN NORMAL 23004 0
2* fscsi0/path2 OPEN NORMAL 133 0
3* fscsi0/path3 OPEN NORMAL 133 0
4* fscsi1/path4 OPEN NORMAL 133 0
5 fscsi1/path5 OPEN NORMAL 33827 0
DEV#: 18 DEVICE NAME: hdisk18 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018086E4080000000000022E
===========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0Solution
I talked to the storage team about it and they investigated the problem from their side. It turns out that there were duplicate aliases defined on the SAN fabrics for both the current and old name of the VIOS (at some point the VIOS had been renamed).
The storage team removed the old aliases and that the additional paths to go into FAILED mode on the VIOS. I then used the diag command to reset the FC adapter:
VIOS:> diag -d fcs0 -vThe command hung for a while and then I aborted it. It also generated a number LINK errors which would indicate zoning(?) problems on the switches:
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
4B436A3D 0622175213 T H fscsi0 LINK ERROR
4B436A3D 0622175113 T H fscsi0 LINK ERROR
4B436A3D 0622175113 T H fscsi0 LINK ERROR
4B436A3D 0622175113 T H fscsi0 LINK ERROR
4B436A3D 0622175013 T H fscsi0 LINK ERROR
4B436A3D 0622175013 T H fscsi0 LINK ERROR
4B436A3D 0622175013 T H fscsi0 LINK ERROR
4B436A3D 0622174913 T H fscsi0 LINK ERROR
4B436A3D 0621165113 T H fscsi0 LINK ERROR
4B436A3D 0621165013 T H fscsi0 LINK ERROR
...
DC73C03A 0621161613 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 0621161613 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 0621161613 T S fscsi0 SOFTWARE PROGRAM ERROR
DE3B8540 0621161513 P H hdisk17 PATH HAS FAILED
DE3B8540 0621161513 P H hdisk14 PATH HAS FAILED
...
However, the FAILED paths did disappear after a while and all paths returned to normal except the ones that were originally linked to the old aliases, e.g.:
VIOS:> lspath -t -l hdisk1
Defined hdisk1 fscsi0 0
Defined hdisk1 fscsi0 1
Enabled hdisk1 fscsi1 2
Enabled hdisk1 fscsi1 3
Enabled hdisk1 fscsi0 4
Enabled hdisk1 fscsi0 5These can be removed as follows for each LUN:
VIOS:> rmpath -l hdisk1 -p fscsi0 -i 0 -d
paths Deleted
VIOS:> rmpath -l hdisk1 -p fscsi0 -i 1 -d
paths Deleted
VIOS:> lspath -t -l hdisk1
Enabled hdisk1 fscsi1 2
Enabled hdisk1 fscsi1 3
Enabled hdisk1 fscsi0 4
Enabled hdisk1 fscsi0 5
Leave a comment