Dead gateway detection can’t ping the last remaining default gateway

Problem

The SYSLOG is showing following messages on some blades:

Apr 12 07:26:37 uxvm552 vmunix: Dead gateway detection can't ping the last remaining default gateway at 0xa5b0b01 .See ndd -h ip_ire_gw_probe for more info
Apr 12 08:02:38 uxvm552 vmunix: Dead gateway detection can't ping the last remaining default gateway at 0xa5b0b01 .See ndd -h ip_ire_gw_probe for more info
Apr 12 08:13:09 uxvm552 vmunix: Dead gateway detection can't ping the last remaining default gateway at 0xa5b0b01 .See ndd -h ip_ire_gw_probe for more info
Apr 12 08:13:09 uxvm552 vmunix: Dead gateway detection can't ping the last remaining default gateway at 0xa5b0b01 .See ndd -h ip_ire_gw_probe for more info

Check the network settings on the host:

# ndd -get /dev/ip ip_ire_status | grep -e IRE_GATEWAY -e flag

This results in a list of all gateways, the flags will indicate a dead gateway.

Another option ip_ire_gw_probe_interval is available which changes the frequency in which such probes will be performed.

The systems have ip_ire_gw_probe enabled, which causes it to ping the default gateway(s) every 3 minutes and disable the gateway if there’s no response. ….

I decided to check the corresponding VC. In the system log we can see:

2012-04-12T07:24:40+02:00 VCEX3C420200CJ vcmd: [PRO:Profile_hpbim006:6013:Minor] Profile state DEGRADED : At least 1 connection OK but not all connections OK, Previous: Enet Network state OK
2012-04-12T07:24:40+02:00 VCEX3C420200CJ vcmd: [VCD:HPCL03-VC:1024:Minor] Domain state DEGRADED : 1+ enclosures & profiles OK, DEGRADED, UNKNOWN, NOT-MAPPED, Previous: Domain state OK
2012-04-12T07:24:40+02:00 VCEX3C420200CJ vcmd: [PRO:Profile_hpbim007:6013:Minor] Profile state DEGRADED : At least 1 connection OK but not all connections OK, Previous: Enet Network state OK
2012-04-12T07:24:40+02:00 VCEX3C420200CJ vcmd: [PRO:Profile_hpbis005:6013:Minor] Profile state DEGRADED : At least 1 connection OK but not all connections OK, Previous: Enet Network state OK
2012-04-12T07:24:40+02:00 VCEX3C420200CJ vcmd: [PRO:Profile_hpbis006:6013:Minor] Profile state DEGRADED : At least 1 connection OK but not all connections OK, Previous: Enet Network state OK
2012-04-12T07:24:40+02:00 VCEX3C420200CJ vcmd: [PRO:Profile_hpbis007:6013:Minor] Profile state DEGRADED : At least 1 connection OK but not all connections OK, Previous: Enet Network state OK
2012-04-12T08:13:46+02:00 VCEX3C420200CJ vcmd: [NET:Profile_hpbim006:7010:Info] Enet Network state OK : All connections, PhysicalServer OK, Previous: Profile state DEGRADED
2012-04-12T08:13:46+02:00 VCEX3C420200CJ vcmd: [NET:Profile_hpbim007:7010:Info] Enet Network state OK : All connections, PhysicalServer OK, Previous: Profile state DEGRADED
2012-04-12T08:13:46+02:00 VCEX3C420200CJ vcmd: [NET:Profile_hpbis005:7010:Info] Enet Network state OK : All connections, PhysicalServer OK, Previous: Profile state DEGRADED
2012-04-12T08:13:46+02:00 VCEX3C420200CJ vcmd: [NET:Profile_hpbis006:7010:Info] Enet Network state OK : All connections, PhysicalServer OK, Previous: Profile state DEGRADED
2012-04-12T08:13:46+02:00 VCEX3C420200CJ vcmd: [NET:Profile_hpbis007:7010:Info] Enet Network state OK : All connections, PhysicalServer OK, Previous: Profile state DEGRADED
2012-04-12T08:13:46+02:00 VCEX3C420200CJ vcmd: [VCD:HPCL03-VC:1020:Info] Domain state OK : All enclosures & profiles OK, Previous: Domain state DEGRADED

Looks like there was a short link interruption (for whatever reason).