Problem

On a VIOS we see a alot of duplicate paths for most of the LUNs:

VIOS:> lspath
status  name     parent connection

Enabled hdisk1   fscsi0 5005076801104c32,0
Enabled hdisk2   fscsi0 5005076801104c32,1000000000000
Enabled hdisk3   fscsi0 5005076801104c32,2000000000000
Enabled hdisk4   fscsi0 5005076801104c32,3000000000000
Enabled hdisk5   fscsi0 5005076801104c32,4000000000000
Enabled hdisk6   fscsi0 5005076801104c32,5000000000000
Enabled hdisk7   fscsi0 5005076801104c32,6000000000000
Enabled hdisk8   fscsi0 5005076801104c32,7000000000000
Enabled hdisk9   fscsi0 5005076801104c32,8000000000000
Enabled hdisk1   fscsi0 5005076801104a2a,0
Enabled hdisk2   fscsi0 5005076801104a2a,1000000000000
Enabled hdisk3   fscsi0 5005076801104a2a,2000000000000
Enabled hdisk4   fscsi0 5005076801104a2a,3000000000000
Enabled hdisk5   fscsi0 5005076801104a2a,4000000000000
Enabled hdisk6   fscsi0 5005076801104a2a,5000000000000
Enabled hdisk7   fscsi0 5005076801104a2a,6000000000000
Enabled hdisk8   fscsi0 5005076801104a2a,7000000000000
Enabled hdisk9   fscsi0 5005076801104a2a,8000000000000
Enabled hdisk0   fscsi0 500507680140dc81,0
Enabled hdisk0   fscsi0 500507680140dc91,0
Enabled hdisk1   fscsi1 5005076801304a2a,0
Enabled hdisk2   fscsi1 5005076801304a2a,1000000000000
Enabled hdisk3   fscsi1 5005076801304a2a,2000000000000
Enabled hdisk4   fscsi1 5005076801304a2a,3000000000000
Enabled hdisk5   fscsi1 5005076801304a2a,4000000000000
Enabled hdisk6   fscsi1 5005076801304a2a,5000000000000
Enabled hdisk7   fscsi1 5005076801304a2a,6000000000000
Enabled hdisk8   fscsi1 5005076801304a2a,7000000000000
Enabled hdisk9   fscsi1 5005076801304a2a,8000000000000
Enabled hdisk1   fscsi1 5005076801304c32,0
Enabled hdisk2   fscsi1 5005076801304c32,1000000000000
Enabled hdisk3   fscsi1 5005076801304c32,2000000000000
Enabled hdisk4   fscsi1 5005076801304c32,3000000000000
Enabled hdisk5   fscsi1 5005076801304c32,4000000000000
Enabled hdisk6   fscsi1 5005076801304c32,5000000000000
Enabled hdisk7   fscsi1 5005076801304c32,6000000000000
Enabled hdisk8   fscsi1 5005076801304c32,7000000000000
Enabled hdisk9   fscsi1 5005076801304c32,8000000000000
Enabled hdisk0   fscsi1 500507680120dc81,0
Enabled hdisk0   fscsi1 500507680120dc91,0
Enabled hdisk1   fscsi0 5005076801204c32,0
Enabled hdisk2   fscsi0 5005076801204c32,1000000000000
Enabled hdisk3   fscsi0 5005076801204c32,2000000000000
Enabled hdisk4   fscsi0 5005076801204c32,3000000000000
Enabled hdisk5   fscsi0 5005076801204c32,4000000000000
Enabled hdisk6   fscsi0 5005076801204c32,5000000000000
Enabled hdisk7   fscsi0 5005076801204c32,6000000000000
Enabled hdisk8   fscsi0 5005076801204c32,7000000000000
Enabled hdisk9   fscsi0 5005076801204c32,8000000000000
Enabled hdisk1   fscsi0 5005076801204a2a,0
Enabled hdisk2   fscsi0 5005076801204a2a,1000000000000
Enabled hdisk3   fscsi0 5005076801204a2a,2000000000000
Enabled hdisk4   fscsi0 5005076801204a2a,3000000000000
Enabled hdisk5   fscsi0 5005076801204a2a,4000000000000
Enabled hdisk6   fscsi0 5005076801204a2a,5000000000000
Enabled hdisk7   fscsi0 5005076801204a2a,6000000000000
Enabled hdisk8   fscsi0 5005076801204a2a,7000000000000
Enabled hdisk9   fscsi0 5005076801204a2a,8000000000000
Enabled hdisk10  fscsi0 5005076801104c32,9000000000000
Enabled hdisk11  fscsi0 5005076801104c32,a000000000000
Enabled hdisk12  fscsi0 5005076801104c32,b000000000000
Enabled hdisk10  fscsi0 5005076801204c32,9000000000000
Enabled hdisk11  fscsi0 5005076801204c32,a000000000000
Enabled hdisk12  fscsi0 5005076801204c32,b000000000000
Enabled hdisk10  fscsi0 5005076801104a2a,9000000000000
Enabled hdisk11  fscsi0 5005076801104a2a,a000000000000
Enabled hdisk12  fscsi0 5005076801104a2a,b000000000000
Enabled hdisk10  fscsi0 5005076801204a2a,9000000000000
Enabled hdisk11  fscsi0 5005076801204a2a,a000000000000
Enabled hdisk12  fscsi0 5005076801204a2a,b000000000000
Enabled hdisk10  fscsi1 5005076801304a2a,9000000000000
Enabled hdisk11  fscsi1 5005076801304a2a,a000000000000
Enabled hdisk12  fscsi1 5005076801304a2a,b000000000000
Enabled hdisk10  fscsi1 5005076801304c32,9000000000000
Enabled hdisk11  fscsi1 5005076801304c32,a000000000000
Enabled hdisk12  fscsi1 5005076801304c32,b000000000000
Enabled sasdisk0 sas0   20000,0
Enabled hdisk13  fscsi0 5005076801104c32,c000000000000
Enabled hdisk14  fscsi0 5005076801104c32,d000000000000
Enabled hdisk15  fscsi0 5005076801104c32,e000000000000
Enabled hdisk16  fscsi0 5005076801104c32,f000000000000
Enabled hdisk17  fscsi0 5005076801104c32,10000000000000
Enabled hdisk13  fscsi0 5005076801204c32,c000000000000
Enabled hdisk14  fscsi0 5005076801204c32,d000000000000
Enabled hdisk15  fscsi0 5005076801204c32,e000000000000
Enabled hdisk16  fscsi0 5005076801204c32,f000000000000
Enabled hdisk17  fscsi0 5005076801204c32,10000000000000
Enabled hdisk13  fscsi0 5005076801104a2a,c000000000000
Enabled hdisk14  fscsi0 5005076801104a2a,d000000000000
Enabled hdisk15  fscsi0 5005076801104a2a,e000000000000
Enabled hdisk16  fscsi0 5005076801104a2a,f000000000000
Enabled hdisk17  fscsi0 5005076801104a2a,10000000000000
Enabled hdisk13  fscsi0 5005076801204a2a,c000000000000
Enabled hdisk14  fscsi0 5005076801204a2a,d000000000000
Enabled hdisk15  fscsi0 5005076801204a2a,e000000000000
Enabled hdisk16  fscsi0 5005076801204a2a,f000000000000
Enabled hdisk17  fscsi0 5005076801204a2a,10000000000000
Enabled hdisk13  fscsi1 5005076801304a2a,c000000000000
Enabled hdisk14  fscsi1 5005076801304a2a,d000000000000
Enabled hdisk15  fscsi1 5005076801304a2a,e000000000000
Enabled hdisk16  fscsi1 5005076801304a2a,f000000000000
Enabled hdisk17  fscsi1 5005076801304a2a,10000000000000
Enabled hdisk13  fscsi1 5005076801304c32,c000000000000
Enabled hdisk14  fscsi1 5005076801304c32,d000000000000
Enabled hdisk15  fscsi1 5005076801304c32,e000000000000
Enabled hdisk16  fscsi1 5005076801304c32,f000000000000
Enabled hdisk17  fscsi1 5005076801304c32,10000000000000
Enabled hdisk18  fscsi0 500507680140dc81,1000000000000
Enabled hdisk18  fscsi0 500507680140dc91,1000000000000
Enabled hdisk18  fscsi1 500507680120dc81,1000000000000
Enabled hdisk18  fscsi1 500507680120dc91,1000000000000

For example: hdisk17 has 6 paths instead of 4:

VIOS:> lspath -dev hdisk17

status  name    parent connection

Enabled hdisk17 fscsi0 5005076801104c32,10000000000000
Enabled hdisk17 fscsi0 5005076801204c32,10000000000000
Enabled hdisk17 fscsi0 5005076801104a2a,10000000000000
Enabled hdisk17 fscsi0 5005076801204a2a,10000000000000
Enabled hdisk17 fscsi1 5005076801304a2a,10000000000000
Enabled hdisk17 fscsi1 5005076801304c32,10000000000000

But yet a newly added hdisk18 has only 4:

VIOS:> lspath -dev hdisk18

status  name    parent connection

Enabled hdisk18 fscsi0 500507680140dc81,1000000000000
Enabled hdisk18 fscsi0 500507680140dc91,1000000000000
Enabled hdisk18 fscsi1 500507680120dc81,1000000000000
Enabled hdisk18 fscsi1 500507680120dc91,1000000000000

The datapath command shows the additional paths also:

VIOS:> datapath query device

Total Dual Active and Active/Asymmetric Devices : 19


DEV#:   1  DEVICE NAME: hdisk1  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E82619000000000002EF6
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path2           OPEN   NORMAL         21          0
    1           fscsi1/path3           OPEN   NORMAL    1719645          0
    2           fscsi0/path0           OPEN   NORMAL    1087376          0
    3*          fscsi0/path1           OPEN   NORMAL         21          0
    4           fscsi0/path4           OPEN   NORMAL    1087492          0
    5*          fscsi0/path5           OPEN   NORMAL         21          0

DEV#:   2  DEVICE NAME: hdisk2  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E82619000000000002FF4
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi1/path2           OPEN   NORMAL    1820666          0
    1*          fscsi1/path3           OPEN   NORMAL         21          0
    2*          fscsi0/path0           OPEN   NORMAL         21          0
    3           fscsi0/path1           OPEN   NORMAL    1156226          0
    4*          fscsi0/path4           OPEN   NORMAL         21          0
    5           fscsi0/path5           OPEN   NORMAL    1154046          0

DEV#:   3  DEVICE NAME: hdisk3  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E82619000000000002FF5
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
	0*          fscsi1/path2           OPEN   NORMAL         35          0
    1           fscsi1/path3           OPEN   NORMAL    1435944          0
    2           fscsi0/path0           OPEN   NORMAL     883921          0
    3*          fscsi0/path1           OPEN   NORMAL         28          0
    4           fscsi0/path4           OPEN   NORMAL     884695          0
    5*          fscsi0/path5           OPEN   NORMAL         35          0

DEV#:   4  DEVICE NAME: hdisk4  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E8261900000000000303E
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path2           OPEN   NORMAL         21          0
    1           fscsi1/path3           OPEN   NORMAL     285190          0
    2           fscsi0/path0           OPEN   NORMAL      98426          0
    3*          fscsi0/path1           OPEN   NORMAL         21          0
	4           fscsi0/path4           OPEN   NORMAL      97737          0
    5*          fscsi0/path5           OPEN   NORMAL         21          0

DEV#:   5  DEVICE NAME: hdisk5  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000030EF
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path2          CLOSE   NORMAL         70          0
    1           fscsi1/path3          CLOSE   NORMAL         80          0
    2           fscsi0/path0          CLOSE   NORMAL         71          0
    3*          fscsi0/path1          CLOSE   NORMAL         70          0
    4           fscsi0/path4          CLOSE   NORMAL         72          0
    5*          fscsi0/path5          CLOSE   NORMAL         70          0

DEV#:   6  DEVICE NAME: hdisk6  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000030F0
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi1/path2          CLOSE   NORMAL         78          0
    1*          fscsi1/path3          CLOSE   NORMAL         70          0
    2*          fscsi0/path0          CLOSE   NORMAL         70          0
    3           fscsi0/path1          CLOSE   NORMAL         71          0
    4*          fscsi0/path4          CLOSE   NORMAL         70          0
    5           fscsi0/path5          CLOSE   NORMAL         74          0

DEV#:   7  DEVICE NAME: hdisk7  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E82619000000000003270
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi1/path2           OPEN   NORMAL     276311          0
    1*          fscsi1/path3           OPEN   NORMAL         21          0
    2*          fscsi0/path0           OPEN   NORMAL         21          0
    3           fscsi0/path1           OPEN   NORMAL      97662          0
    4*          fscsi0/path4           OPEN   NORMAL         21          0
    5           fscsi0/path5           OPEN   NORMAL      97489          0

DEV#:   8  DEVICE NAME: hdisk8  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000036B8
=========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path2           OPEN   NORMAL         21          0
    1           fscsi1/path3           OPEN   NORMAL     303137          0
    2           fscsi0/path0           OPEN   NORMAL     125066          0
    3*          fscsi0/path1           OPEN   NORMAL         28          0
    4           fscsi0/path4           OPEN   NORMAL     126426          0
    5*          fscsi0/path5           OPEN   NORMAL         21          0

DEV#:   9  DEVICE NAME: hdisk9  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000037D1
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path2           OPEN   NORMAL         21          0
    1           fscsi1/path3           OPEN   NORMAL     308114          0
    2           fscsi0/path0           OPEN   NORMAL     124842          0
    3*          fscsi0/path1           OPEN   NORMAL         21          0
    4           fscsi0/path4           OPEN   NORMAL     124216          0
    5*          fscsi0/path5           OPEN   NORMAL         21          0

DEV#:   0  DEVICE NAME: hdisk0  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018086E40800000000000074
==========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0           OPEN   NORMAL    7093852          0
    1*          fscsi0/path1           OPEN   NORMAL         52          0
    2           fscsi1/path2           OPEN   NORMAL    7385644          0
    3*          fscsi1/path3           OPEN   NORMAL         52          0

DEV#:  10  DEVICE NAME: hdisk10  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000046E4
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path4           OPEN   NORMAL         28          0
    1           fscsi1/path5           OPEN   NORMAL    6991560          0
    2           fscsi0/path0           OPEN   NORMAL    4964095          0
    3           fscsi0/path1           OPEN   NORMAL    4963196          0
    4*          fscsi0/path2           OPEN   NORMAL         28          0
    5*          fscsi0/path3           OPEN   NORMAL         28          0

DEV#:  11  DEVICE NAME: hdisk11  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E82619000000000004A00
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path4           OPEN   NORMAL         28          0
    1           fscsi1/path5           OPEN   NORMAL  735702974          0
    2           fscsi0/path0           OPEN   NORMAL  445782223          0
    3           fscsi0/path1           OPEN   NORMAL  445781860          0
    4*          fscsi0/path2           OPEN   NORMAL         28          0
    5*          fscsi0/path3           OPEN   NORMAL         28          0

DEV#:  12  DEVICE NAME: hdisk12  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E82619000000000004A02
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi1/path4           OPEN   NORMAL         21          0
    1           fscsi1/path5           OPEN   NORMAL   20965908          0
    2           fscsi0/path0           OPEN   NORMAL   13563901          0
    3           fscsi0/path1           OPEN   NORMAL   13564590          0
    4*          fscsi0/path2           OPEN   NORMAL         21          0
    5*          fscsi0/path3           OPEN   NORMAL         21          0

DEV#:  13  DEVICE NAME: hdisk13  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000075CA
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0           OPEN   NORMAL     893304          0
    1           fscsi0/path1           OPEN   NORMAL     851758          0
    2*          fscsi0/path2           OPEN   NORMAL        133          0
    3*          fscsi0/path3           OPEN   NORMAL        133          0
    4*          fscsi1/path4           OPEN   NORMAL        133          0
    5           fscsi1/path5           OPEN   NORMAL    1236751          0

DEV#:  14  DEVICE NAME: hdisk14  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000075CF
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi0/path0           OPEN   NORMAL         56          0
    1*          fscsi0/path1           OPEN   NORMAL         56          0
    2           fscsi0/path2           OPEN   NORMAL     196677          0
    3           fscsi0/path3           OPEN   NORMAL     155511          0
    4           fscsi1/path4           OPEN   NORMAL     211308          0
    5*          fscsi1/path5           OPEN   NORMAL         56          0

DEV#:  15  DEVICE NAME: hdisk15  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000075CC
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0           OPEN   NORMAL    8844974          0
    1           fscsi0/path1           OPEN   NORMAL    8795463          0
    2*          fscsi0/path2           OPEN   NORMAL        133          0
    3*          fscsi0/path3           OPEN   NORMAL        133          0
    4*          fscsi1/path4           OPEN   NORMAL        133          0
    5           fscsi1/path5           OPEN   NORMAL   13549327          0

DEV#:  16  DEVICE NAME: hdisk16  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000075CD
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0*          fscsi0/path0           OPEN   NORMAL        133          0
    1*          fscsi0/path1           OPEN   NORMAL        133          0
    2           fscsi0/path2           OPEN   NORMAL      73812          0
    3           fscsi0/path3           OPEN   NORMAL      31740          0
    4           fscsi1/path4           OPEN   NORMAL      43271          0
    5*          fscsi1/path5           OPEN   NORMAL        133          0

DEV#:  17  DEVICE NAME: hdisk17  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018E826190000000000075CE
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0           OPEN   NORMAL      64733          0
    1           fscsi0/path1           OPEN   NORMAL      23004          0
    2*          fscsi0/path2           OPEN   NORMAL        133          0
    3*          fscsi0/path3           OPEN   NORMAL        133          0
    4*          fscsi1/path4           OPEN   NORMAL        133          0
    5           fscsi1/path5           OPEN   NORMAL      33827          0

DEV#:  18  DEVICE NAME: hdisk18  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018086E4080000000000022E
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0          CLOSE   NORMAL          0          0
    1           fscsi0/path1          CLOSE   NORMAL          0          0
    2           fscsi1/path2          CLOSE   NORMAL          0          0
    3           fscsi1/path3          CLOSE   NORMAL          0          0

Solution

I talked to the storage team about it and they investigated the problem from their side. It turns out that there were duplicate aliases defined on the SAN fabrics for both the current and old name of the VIOS (at some point the VIOS had been renamed).

The storage team removed the old aliases and that the additional paths to go into FAILED mode on the VIOS. I then used the diag command to reset the FC adapter:

VIOS:> diag -d fcs0 -v

The command hung for a while and then I aborted it. It also generated a number LINK errors which would indicate zoning(?) problems on the switches:

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
4B436A3D   0622175213 T H fscsi0         LINK ERROR
4B436A3D   0622175113 T H fscsi0         LINK ERROR
4B436A3D   0622175113 T H fscsi0         LINK ERROR
4B436A3D   0622175113 T H fscsi0         LINK ERROR
4B436A3D   0622175013 T H fscsi0         LINK ERROR
4B436A3D   0622175013 T H fscsi0         LINK ERROR
4B436A3D   0622175013 T H fscsi0         LINK ERROR
4B436A3D   0622174913 T H fscsi0         LINK ERROR
4B436A3D   0621165113 T H fscsi0         LINK ERROR
4B436A3D   0621165013 T H fscsi0         LINK ERROR
...
DC73C03A   0621161613 T S fscsi0         SOFTWARE PROGRAM ERROR
DC73C03A   0621161613 T S fscsi0         SOFTWARE PROGRAM ERROR
DC73C03A   0621161613 T S fscsi0         SOFTWARE PROGRAM ERROR
DE3B8540   0621161513 P H hdisk17        PATH HAS FAILED
DE3B8540   0621161513 P H hdisk14        PATH HAS FAILED
...

However, the FAILED paths did disappear after a while and all paths returned to normal except the ones that were originally linked to the old aliases, e.g.:

VIOS:> lspath -t -l hdisk1

Defined hdisk1 fscsi0 0
Defined hdisk1 fscsi0 1
Enabled hdisk1 fscsi1 2
Enabled hdisk1 fscsi1 3
Enabled hdisk1 fscsi0 4
Enabled hdisk1 fscsi0 5

These can be removed as follows for each LUN:

VIOS:> rmpath -l hdisk1 -p fscsi0 -i 0 -d
paths Deleted

VIOS:> rmpath -l hdisk1 -p fscsi0 -i 1 -d
paths Deleted

VIOS:> lspath -t -l hdisk1               
Enabled hdisk1 fscsi1 2
Enabled hdisk1 fscsi1 3
Enabled hdisk1 fscsi0 4
Enabled hdisk1 fscsi0 5

Comments