[Novalug] LVM vs Partitions

Bryan J. Smith b.j.smith@ieee.org
Mon Mar 15 20:14:46 EDT 2010


The problem with benchmarking software RAID-5 via MD is that you are _always_
buffering writes.  Even exceeding the memory size won't show this.  It's completely
async flush.  Hardware RAID makes part of that a fully sync flush.  Software MD
will _always_ win Bonnie++ benchmarks, _always_.  There's no way to prevent this.
Especially in how Bonnie++ operates.

The only way to benchmark software v. hardware RAID-5/6 is to saturate the
system with service I/O while you are hammering the disks.  Then the off-load
will show up.  The ramming of multiple copies of data up the system interconnnect
just to do an XOR -- which is wholly programmed I/O (PIO) -- will show how the
general purpose CPU makes for a horrendous I/O processor when it could be
otherwise utilized much better.  If you don't have services that saturate I/O due to
services, then software RAID-5 can work well.

Software RAID-5 works well when you write in moderation and don't have I/O contention.
Software RAID-5 is also fine when you're reading heavily, because it's acting just like
a RAID-0 stripe (minus one disk in RAID-5, two in RAID-6).  In fact, Software RAID-5
can be far more ideal here.

Also, the bottleneck is _not_ the channel controller.  The dumbest ATA can blow away
everything with non-blocking I/O.  Native Command Queuing (NCQ) can help the
CPU off-load basic DMA block operations at the controller level.

The problem is turning the CPU into an I/O processor by ramming duplicate copies
of its data through its interconnect just for a LOAD-STO.  That's the problem with any
generic processor or microcontroller.  However, if your interconnect has a directly
attached ASIC that is designed to do XORs in-line, and is just commanded by a
microcontroller to do that, then it's a true "storage switch" that replicates data and
computes XORs (let alone verifies them on reads, which is optional, but a nice
addition that many do) at block transfer speeds.

-- Bryan

P.S.  Software RAID-1 actually doesn't need to bother the CPU.  It just does two
commits of the same block from memory via DMA mechanisms.  Not XOR is
required.  Although a hardware storage switch that compares mirrored data is
also an ideal.  LVM mirroring actually uses a mirror log that journals differences
as well for comparison, separate from the data copies itself.  That is both a nice
advantage as well as an added detail that has to be accomodated.


 -- 
Bryan J  Smith           Professional, Technical Annoyance 
Linked Profile:         http://www.linkedin.com/in/bjsmith 
---------------------------------------------------------- 
Red Hat:  That 'other' American software company built on
open customer selection of options and value, instead of
controlled distribution channels of forced bundle and lock



----- Original Message ----
From: Bryan Seitz <seitz@bsd-unix.net>

This didn't seem to be the case with my 2.2Ghz Core2Duo and 6 disks in Raid5 :)
In fact, my bonnie++ benchmark beat some hardware raid5 (HP) at work :)

20MiBps, really???  There are many factors which can play into the performance
of software raid like:

* The IDE/Sata chipset the disks are attached to
* The RPM of the disks, the cache size of the disks, the platter density of the disks

I've definitely seen a lot higher throughput with MD Raid-5, although I don't have any
systems currently using it since FreeBSD started supporting ZFS.



More information about the Novalug mailing list