And sometime between 4:30am (or so) and 8:00am, something else must
have happened because we have received emails and calls from a couple
customers of problems with their virtual machines.
Not a single message in the logs on the NAS about any error, warning,
or critical issue occuring. All logs are clean with normal notices of
informational standing (heartbeats being received, connections for
SNMP, etc).
The VMware logs tell a different story and show that at 5:48am there
was a problem, with recovery at 6:04am.
There are only ~15 VMs left on this volume and I'll be moving them
during the business day today.
I have some new, temporary, storage ready to get mounted up.
We are (Doug and I) going to get everyone off of the storage
completely (and onto the storage mentioned above) so we can take the
hardware apart and try to figure out what is going on. This has to be
a hardware issue, the problem follows the volume, the server operating
system continues to run, is responsive.
All VMs will be removed from this storage cluster so we can start to
tear into the hardware ASAP.
First priority is to remove all VMs from volume01 onto other storage.
This is in process.
Second priority is to remove all VMs from volume02 onto other storage.
This will occur as the day progresses.
We have a few pieces of storage already online. Approximately 1/2 of
it is clustered storage and will be utilized immediately (it was the
NetApp gear we were using before this new storage cluster was brought
online in May, 2010 and is still quite serviceable). The other 1/2
are servers were we have swapped out the normal hard disk drives for
larger ones, installed a server OS to handle NFS NAS duties, but these
systems are *not* clusterable and will only be used for a very short
period of time while we work out the issues with our production NAS.
Support can be reached Monday thru Friday from 8:00am until 8:00pm via
phone at 612-337-6340, or via email at [log in to unmask]
--
Mike Horwath ipHouse - Welcome home! [log in to unmask]
The universe is an island, surrounded by whatever it is
that surrounds universes. - Berkeley Fortune
