[Novalug] LVM vs Partitions
Bryan J. Smith
b.j.smith@ieee.org
Mon Mar 15 20:14:46 EDT 2010
The problem with benchmarking software RAID-5 via MD is that you are _always_
buffering writes. Even exceeding the memory size won't show this. It's completely
async flush. Hardware RAID makes part of that a fully sync flush. Software MD
will _always_ win Bonnie++ benchmarks, _always_. There's no way to prevent this.
Especially in how Bonnie++ operates.
The only way to benchmark software v. hardware RAID-5/6 is to saturate the
system with service I/O while you are hammering the disks. Then the off-load
will show up. The ramming of multiple copies of data up the system interconnnect
just to do an XOR -- which is wholly programmed I/O (PIO) -- will show how the
general purpose CPU makes for a horrendous I/O processor when it could be
otherwise utilized much better. If you don't have services that saturate I/O due to
services, then software RAID-5 can work well.
Software RAID-5 works well when you write in moderation and don't have I/O contention.
Software RAID-5 is also fine when you're reading heavily, because it's acting just like
a RAID-0 stripe (minus one disk in RAID-5, two in RAID-6). In fact, Software RAID-5
can be far more ideal here.
Also, the bottleneck is _not_ the channel controller. The dumbest ATA can blow away
everything with non-blocking I/O. Native Command Queuing (NCQ) can help the
CPU off-load basic DMA block operations at the controller level.
The problem is turning the CPU into an I/O processor by ramming duplicate copies
of its data through its interconnect just for a LOAD-STO. That's the problem with any
generic processor or microcontroller. However, if your interconnect has a directly
attached ASIC that is designed to do XORs in-line, and is just commanded by a
microcontroller to do that, then it's a true "storage switch" that replicates data and
computes XORs (let alone verifies them on reads, which is optional, but a nice
addition that many do) at block transfer speeds.
-- Bryan
P.S. Software RAID-1 actually doesn't need to bother the CPU. It just does two
commits of the same block from memory via DMA mechanisms. Not XOR is
required. Although a hardware storage switch that compares mirrored data is
also an ideal. LVM mirroring actually uses a mirror log that journals differences
as well for comparison, separate from the data copies itself. That is both a nice
advantage as well as an added detail that has to be accomodated.
--
Bryan J Smith Professional, Technical Annoyance
Linked Profile: http://www.linkedin.com/in/bjsmith
----------------------------------------------------------
Red Hat: That 'other' American software company built on
open customer selection of options and value, instead of
controlled distribution channels of forced bundle and lock
----- Original Message ----
From: Bryan Seitz <seitz@bsd-unix.net>
This didn't seem to be the case with my 2.2Ghz Core2Duo and 6 disks in Raid5 :)
In fact, my bonnie++ benchmark beat some hardware raid5 (HP) at work :)
20MiBps, really??? There are many factors which can play into the performance
of software raid like:
* The IDE/Sata chipset the disks are attached to
* The RPM of the disks, the cache size of the disks, the platter density of the disks
I've definitely seen a lot higher throughput with MD Raid-5, although I don't have any
systems currently using it since FreeBSD started supporting ZFS.
More information about the Novalug
mailing list