The storage cluster is back online!
Why so long since my last update?
Took quite a while to get all customers off of the cluster and to do
so I had to create many temporary storage systems. (A total of 6
different systems were used to as the destination for all of the data
from the storage cluster.)
A personal thanks from me to our larger Enterprise Hosting on VMware
customer base for helping move their data onto this temporary storage.
I appreciated the ability to work with other clients while the data
was being moved. THANK YOU!
Now things are back and I am continuing testing and initial production
work on the revamped/rebuilt cluster.
I have recreated the volumes from scratch and beat them up already,
now it is time to move a little closer to full production.
Things are re-mounted onto the different VMware physical hosts and I
have started moving over some internal servers to continue testing.
I'll be working with the larger clients again to help with movement of
their data as production goes live in the coming days.
Now, what changed?
A) all SSDs are gone - and probably will never return. Using SATA
SSDs with SAS interposers were creating some problems that were hard
to recreate and track down, once they were removed .. everything
stabilized and no problems were seen again. But I still did not
trust things and that is why everything was rebuilt, more below.
B) the SAS host-bus-adapters (HBAs) in the controller units were
swapped out for newer models. The LSI 1068E controllers worked just
fine BUT disk removal and insertion notifications were not happening
to the underlying operating system. I tested this out on
OpenSolaris, FreeBSD 8, and Ubuntu 10.04 LTS. Yes, some notices
showed up in the logs, but the notices were 'error code 0x18' or
'event id 0x12 occurred'. These are not very helpful notifications.
C) SAS HBAs are running current revision of their BIOS and firmware
D) all cabling was tested as best as I could. I pushed ~4 TB of data
today alone testing things out.
Support can be reached Monday thru Friday from 8:00am until 8:00pm via
phone at 612-337-6340, or via email at [log in to unmask]
--
Mike Horwath ipHouse - Welcome home! [log in to unmask]
The universe is an island, surrounded by whatever it is
that surrounds universes. - Berkeley Fortune
