[SATLUG] Trying to do some Fibre Channel Expirementation

David Kowis dkowis at shlrm.org
Fri May 28 08:12:09 CDT 2010

First some background:
I've got a single computer at home that I use to experiment with running
xen. It's a pretty beefy server, and runs about 10 VMs right now. The
storage on it is currently a mdadm raid5 array of 3x 1tb drives. The
performance of which isn't doing so well, since the workload is ill
suited to raid5. And then there's the write hole for when the power goes
out. A hardware raid5 card will mostly solve the problem but is like
$400. What I want to do is use a opensolaris based server to serve up
storage over a FC connection. Then I get the benefits of ZFS, which
brings so much more to the table than a hardware raid5 card does and for
only a little bit more money.

The plan is to use osol to run a zfs based storage machine, with the
block level deduplication (which will save me plenty of space with
virtual machines) running raidz and connected to my xen server over
fibre channel. I can use COMSTAR in osol to export the zvols over the FC
connection, and then pick them up in the Xen box as drives and just use
them. I'll have my storage separated from my hypervisor, which I've been
told is 'good karma' since first I tried to use osol as my domain0.
Unfortunately, since Oracle's acquisition of Sun, the osol
implementation of Xen has not been developed or anything. I opted to
avoid that, and stick with the linux xen, so getting a separate server
for storage is the only way to go. Using ZFS gets me several benefits
that I don't have:
	* much faster disks, since raidz is much smarter than raid5
	* Automatic bitrot detection
	* block-level deduplication
Resilvering, block-level deduplication, and raidz are worth the extra
money. Especially since I'm a hobbyist, and I can't afford advanced
storage arrays with magical software to check the data consistency, and
raidz is free and will work with commodity disks (so long as the
hardware is new enough to be supported (SATA is)).

However, OpenSolaris' hardware support is crap, so I'm having trouble
testing my setup. I have an oldish dell 2400 (I think that's the number)
that has 64-bit PCI slots. Unfortunately, osol doesn't support the scsi
controller in there, so I can't actually install osol onto it, so I
cannot test anything. I have tried to put the 64-bit pci cards into a
32-bit slot, and that doesn't work. The system locked up when it tried
to initialize the card.

What I'm wondering is if anyone has a computer that has 64-bit pci slots
that I could borrow for a while to see if I could get this working. I
have other hardware that I'd be willing to trade for it, or if you're
willing to just let me borrow it, that'd be awesome too :)

I'd rather not spend $500 to buy the components to build this storage
server, and then find out that it's not possible, hardware wise. I guess
I could just spend more money and purchase FC cards that I know work,
but I'm trying to operate on a budget.

Thanks for your time,

More information about the SATLUG mailing list