[Novalug] best hard disk setup for home file server?

Bryan J. Smith b.j.smith@ieee.org
Tue Oct 20 16:27:43 EDT 2009


Well, I don't know if I'd argue that symlinks are viable.
I was talking more mounts than symlinks.

My standard filesystem[1] for static data is ...
  /home/mobile/Static

And my Photos go under ...
  /home/mobile/Static/Photos

Over the last five (5) years, I've shot a good TiB of photos.
That's not something I want on a single filesystem, much less
mixed in with a lot of other files, even if they are all static.

I always organized my photos by type -- Travel for general
travel, UCF for photos taken at my Alma Mater's football
games, etc...  Underneath those are years and places.

It wasn't even two (2) years into shooting that half of my photos
were a result of 2-3,000 shots at every football game (I travel to
a good ten (10) games every year -- all home and most away
games), that it was 200-300GB/season.  

So I created a new filesystem[1] and mounted it accordingly
when I started running into over 256GiB of photos:  
  /home/mobile/Static/Photos/UCF

Just this year, because the sizes are 250GB+/season, I decided
to create a new filesystem[1] for:  
  /home/mobile/Static/Photos/UCF/2009

The years of 2005, 2006, 2007 and 2008 might have their own
directories, but they are on the original Static/Photos/UCF filesystem.
The 2009 directory is in its own, new, UCF/2009 filesystem.  The
user doesn't see any different at all, be it over HTTP, NFS, SFTP or
SMB.

I can grow filesystems as necessary, create new ones when I decide
I want to localize more storage, etc...  These filesystems can even be
on different VGs, but mounted in the same area.  I can use LVM
commands to move things around too.

-- Bryan

[1] To be really accurate, they don't even need to be on the same
server.  Why would I want such?  Well, I have two "home" networks
separated by 1,000 miles.  ;)

To start, the actual, real, original filesystem is ...
  /export/dilbert51/Static
Which automounts (NFS, Bind, etc...) to:  
  /home/mobile/Static

My own, personal/professional, longstanding convention for data
has been:  
  /export/(server)/(type)
With the automount:  
  /home/(domain)/(type)

There are countless reasons for this approach, but it breaks down
these reasons:  

- The "source" filesystem has a path with the servername in it, which allows
autofs to differentiate if it's local (and uses a bind mount) or remote (and
uses NFS or another mount)

- The "clients" use a path with the domain name in it, because automounter
maps are usually defined per domain (be it NIS, NIS+, LDAP, Winbindd,
custom schema in AD-LDAP, etc...).

In my case, I have two domains, "Oviedo" and "Mobile."  Oviedo is Oviedo,
FL, where my permanent residence is.  Mobile is my domain for traveling,
including my current Maryland residence.

And yes, this allows me to do all sorts of things.  Just because I hit
/home/oviedo/Static/Photos/UCF doesn't mean I hit an Oviedo server.
It just means I'm just using the automounter map for Oviedo.  I have less
storage in Oviedo than my Mobile, so I could define that even Oviedo
servers hit the Mobile server.

Say, for example, I only define ./UCF/2009 as hitting the Oviedo server
because I have more storage on the Mobile server and my wife doesn't
hit my photos from years ago as much as she does current stuff.  So I'm
only rsync'ing photos from this year back to Oviedo, and not 2008 or
earlier.

So my Oviedo automounter map has 2009 defined to dilbert.oviedo while
all other UCF/200x mounts hit to dilber51.mobile (across the VPN).


 -- 
Bryan J  Smith           Professional, Technical Annoyance 
Linked Profile:         http://www.linkedin.com/in/bjsmith 
---------------------------------------------------------- 
Red Hat:  That 'other' American software company built on
open customer selection of options and value, instead of
controlled distribution channels of forced bundle and lock



----- Original Message ----
From: James Ewing Cottrell 3rd <JECottrell3@comcast.net>
To: Richard Ertel <richard.ertel@gmail.com>
Cc: Bryan J. Smith <b.j.smith@ieee.org>; novalug-bounces@calypso2.tux.org; Novalug <novalug@calypso2.tux.org>
Sent: Tue, October 20, 2009 2:50:59 PM
Subject: Re: [Novalug] best hard disk setup for home file server?

i was arguing that you really don't care where the actual movies go. you are going to build an index to them elsewhere and symlink to the actual movie/tv show. heck, build several indices, a poor man's database.

as for space, if you had a 1TB space carved up into ten 100G partitions, and each movie was an average of 2G, you'd end up wasting 10 x 2G / 2 or 10G on the average. ok, that's five movies, but it's just 1% of your space, and only makes a difference if you want to use it all.

as they say, don't put all your eggs in one basket.

jim

Richard Ertel wrote:
> i have a movies directory, then inside that, each movie is in it's own
> directory. ditto with tv shows, each series has it's own directory
> inside the "tv shows" directory.
> 
> my main interest in keeping one big filesystem is effecient use of
> space. if i have separate filesystems for each kind of media, then
> what happens when i load up proportionately more movies than tv shows?
> i might run out of space on the movies filesystem, but still have
> plenty of room in tv shows. feels more complicated from an end-user
> standpoint, but i see the argument for it being technically more
> sound.
> 
> On Fri, Oct 16, 2009 at 12:39, James Ewing Cottrell 3rd
> <JECottrell3@comcast.net> wrote:
>> Richard Ertel wrote:
>>> i prefer one volume group because that's the most efficient way to
>>> store a bunch of files. if i have 2 TB of movies, for example, but two
>>> 1.5 TB volumes, then i have to split the group somehow. seems messy to
>>> me.
>>> 
>>> are there better solutions?
>> What, you're gonna put them all in one directory? If you put them in
>> separate directories, like scifi (oops, syfy?), action, western, comedy,
>> drama, pron, etc .....
>> 
>> then you can just use smaller partitions and build your tree via mounts.
>> 
>> heck, with stuff that big, just stuff them in a next available partition and
>> build a tree of symlinks to them...that way you can index them by year,
>> director, actor, as many ways as you like.
>> 
>> JIM
>> 
>> P.S. Yes, I think you should ditch those 1.5 TBs ... I'd be glad to recycle
>> them for you.
>> 
> 
> 
> ------------------------------------------------------------------------
> 
> 
> No virus found in this incoming message.
> Checked by AVG - www.avg.com Version: 8.5.421 / Virus Database: 270.14.20/2440 - Release Date: 10/16/09 06:32:00
> 



More information about the Novalug mailing list