On Monday, Oct 10th 2016, starting at 8:59AM, one of our main NetApp filers
dedicded to die on its own. Even though it is a redundant system that
the other controller should have taken over, for reasons unknown yet
it decided it couldn't. Since the 2nd controller couldn't take it over
everything it decided to reboot to try to fix the problem as well. At
9:15AM it was hard power cycled and everything came online immediately
after.
This filer holds about half the data for Email and Web sites for much
of our hosting options, but would have locked up those systems trying
to access data no longer accessible. Email in and out would have been
deferred during this period, but still delivered and received after
everything was up. Many hosted web sites here would have been
inaccessible during this period, but came back up after the problem cleared.
So everything is at 100% now.
After the power cycle, everything came online, and we are checking through
all our systems to make sure they are working correctly.
We'll also make sure that the failover is working correctly on this
filer, but it is probably unknown as to what actually caused the major
panic and lockup. We've used these filers for dozens of years without
a major incident like this. No data was lost during this period, only
deferred until things were online again.
If you have any further problems or questions please let us know at
[log in to unmask]
Thank you.
--
Doug McIntyre <[log in to unmask]>
-- ipHouse/Goldengate/Bitstream/ProNS --
Network Engineer/Provisioning/Jack of all Trades
