OpenVZ breaks networking

I planned a reboot of my server(s) over the weekend to fix a hanging ILO problem. Since the remote console was out of order, I had to go the datacenter offering the co-location services and execute the reboot on-site. After powering off, reconnecting the power cables and powering on, the ILO console came back available again so it appeared my problems were fixed. But when my server was still unreachable via SSH after 5 minutes into the boot process, I got suspicious and took a second look at the console. I quickly noticed that none of the network related services and daemons were starting up. The shorewall (firewall), bacula (backup) and bigsister (monitoring) all failed to start. Trying to start Shorewall manually yielded this error message:

can't initialize iptables table 'nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

Since I use DNAT & SNAT on the hardware node of my OpenVZ server, I do need NAT to be available. After tinkering around with some kernel modules for a while and plowing through available log files, Google finally came to the rescue: http://serverfault.com/questions/593263/iptables-nat-does-not-exist

It appeared that since the previous reboot, OpenVZ has been upgraded from v4.6 to v4.7 with an important major change involved ( https://openvz.org/Download/vzctl/4.7/changes):

Disable conntrack for VE0 by default

IP conntrack functionality has some negative impact on venet performance (uo to about 10%), so they better be disabled by default.

Switching following options back on fixed the problem:

# more /etc/modprobe.d/openvz.conf                                                                                   
options nf_conntrack ip_conntrack_disable_ve0=0

Some lessons learned:

• Attended YUM upgrades are great but core software still needs to be checked for important changes (and be thorough enough).
• Changes to kernel-related software can mean that the next reboot will behave differently than before.
• Do not trust 3rd party software suppliers not to break their own software and/or program behaviour from one version to the next.