[Novalug] Considering NAS

Jared Scott jared@jaredscott.com
Mon Sep 9 02:04:00 EDT 2013


SuperMicro isn't exactly cheap, but it's also not the most expensive. A
bunch of companies that sell servers use supermicro as their server and
just change the faceplate. The one that I have is actually one of those and
it had all of its drives removed before I got it. I'm not sure I would have
picked it if I were paying retail. I got it for free as it was being
discarded from my previous job.

It's been very stable. It's survived many power outages (that lasted longer
than my batteries allowed). I've upgrade from FreeBSD 8 to 9, which allowed
me to update the ZFS and ZPOOL versions. I take hourly, daily, weekly, and
monthly snapshots. But I've only used them once, during testing, to see how
I would restore should I need it (I should probably do that again now that
I've thought of it). The script I use is here (
http://andyleonard.com/2010/04/07/automatic-zfs-snapshot-rotation-on-freebsd/
)

I spent a lot of time reading and testing ZFS before trusting my data on it
and I still don't trust it entirely. I could lose 3 drives in 1 vdev before
I have a chance to resilver (rebuild).  It's highly unlikely, but if you
really care about the data make sure you have it in another place
(crashplan, backblaze, etc.)

Another thing to remember when doing ZFS, you want to hand ZFS the raw
drives, not a RAID group. Buy an HBA rather than a RAID controller to hook
up the drives. I like the LSI ones because the support for FreeBSD and
Linux is always there and the cards have been stable for me.

I would also recommend ECC memory, I'm not going to try to explain it
myself you can read about it here: (
http://hardforum.com/showthread.php?t=1689724)

There are ideals for number of drives per vdev based on the raid type you
choose RAIDZ, RAIDZ-2, or RAIDZ-3. This thread explains that and also
helped me understand the relationship of drives, to vdevs, to pools (what
I've been calling a volume) and then finally to the file system: (
http://forums.freebsd.org/showthread.php?t=4641)

You entire volume is only as redundant as the least redundant vdev in it.
Because ZFS spreads the data out across the vdevs that are part of the
Volume. So if you put a single drive as a vdev into a volume that has two
RAIDZ-2 vdevs and you lose that drive, you will lose data.

Again my apologies, if this post is even more confusing than my previous
one, but hopefully it will help point you in the right direction doing your
research.



On Mon, Sep 9, 2013 at 1:33 AM, Christopher Jones <
christopher.donald.jones@gmail.com> wrote:

> Thanks Jared,
>     This answer was very helpful. I was considering zfs myself but wasn't
> sure how easy it was to expand at a later date. ( need to read up on zfs
> more I know). I'm wondering how much something like this will cost. As in..
> whats a reasonable cost for what I'm looking for? It doesn't have to be
> state of the art but I want it to be solid. I'll look into the company you
> mentioned above as well. Has it been stable for you?
>
>
> On Mon, Sep 9, 2013 at 1:17 AM, Jared Scott <jared@jaredscott.com> wrote:
>
>> I use a supermicro 3u chassis with 16 - Hot Swap 3.5" drive bays, and 1
>> internal 2.5 drive bay. I boot off of a 2.5" - 64 GB - SSD into FreeBSD. I
>> have 12 - 2 TB drives, in two 6-drive, RAIDZ-2, vdevs that make up one 13TB
>> volume. I then export that via CIFS and NFS. The NFS I used from my other
>> *nix machines and the CIFS is for when I need to access it via Windows
>> (haven't had much luck with the windows NFS client).
>>
>> There are a bunch of options on how to expand the volumes. The easiest is
>> just to add an additional vdev. So when I approach the 75% full mark I will
>> likely add another 6 drive vdev but with 4 TB drives (or whatever size is
>> the best price when I reach my threshold).
>>
>> When I built this (3 years ago) the support for ZFS on Linux was pretty
>> abysmal. It has since (as I understand it) become quite nice. If I were to
>> do it again, I would probably start with it on linux and test the crap out
>> of it. Assuming it performed well I would use that.
>>
>> ZFS supports dedupe, compression and snapshots natively. I don't use the
>> dedupe because the data I store won't benefit from it. I can emulate time
>> machine on it so I backup my mac to it directly. My wife copies all of her
>> photos to it manually and I use crashplan on her laptop as well (can't have
>> too many copies of the kids photos and videos).
>>
>> Using the array over the lan via NFS should work, but it will depend on
>> how many VM's you run at a time and what the IOPS requirements of them are.
>> I run multiple VM's at home from an older Dell server that sits above the
>> NAS. I run esxi and I connect to it via NFS. I have it perform fine for up
>> to 16 running VM's, but I've also managed to make it run horribly with only
>> 1 VM. It just depends on what you are doing.
>>
>> I know that wasn't exactly what you asked, but maybe you will find it
>> helpful.
>>
>>
>>
>>
>> On Mon, Sep 9, 2013 at 12:42 AM, Christopher Jones <
>> christopher.donald.jones@gmail.com> wrote:
>>
>>> I'm considering buying a 12 bay nas unit. I'm looking for suggestions on
>>> make, if its safe to buy refurbished units (not drives), what to look for
>>> in a good unit.
>>>
>>> I want to use this for backups + if possible I'd like it to house the
>>> logical volumes I use for my kvm's and have them mount from a different
>>> machine.
>>>
>>> What sort of software does the list have experience with / recommend?
>>>
>>> I want to be able to scale this. I'd like to start with 4 or 5 drives
>>> and build up as I need (some way that doesn't require rebuilding the whole
>>> aray). I'd like to have deduplication and snapshots. If possible. I also
>>> want hot swappable drives / power units supplies.
>>>
>>> Any idea what this would cost? Will the delay over the network be to
>>> much for operating systems? Is NAS even capable of it?
>>> Why do i see some of these on ebay for 200 bucks? Is that even possible
>>> or is that a sign its a bad deal??
>>>
>>> Lots of questions I know!! Thanks All!!
>>>
>>> --
>>> Chris Jones
>>> RHCSA
>>>
>>>
>>> _______________________________________________
>>> Novalug mailing list
>>> Novalug@calypso.tux.org
>>> http://calypso.tux.org/mailman/listinfo/novalug
>>>
>>>
>>
>
>
> --
> Chris Jones
> RHCSA
>
>
> _______________________________________________
> Novalug mailing list
> Novalug@calypso.tux.org
> http://calypso.tux.org/mailman/listinfo/novalug
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.firemountain.net/pipermail/novalug/attachments/20130909/ab653b31/attachment.htm>


More information about the Novalug mailing list