# Best (Cheap) RAID / Storage Solution?



## Vautrin (Sep 13, 2008)

Hi,

So I've done it, I've filled up my computer's hard drive with photos.

I'm going to buy a USB RAID, probably around 1TB, so that I'll have plenty of room (at least for a year or two).

Does anyone have any recommendations on good raid / storage solutions?

Thanks,

Dan


----------



## Moglex (Sep 13, 2008)

For storage solutions generally I'd recommend simple USB external hard drives.

They are now absurdly cheap and you can buy three and use them on a grandfather/father/son system which is generally more resilient than raid, especially if you keep the drives in physically separate locations.

By the time you fill up set of 750GB drives it will probably be time to replace them and a typical drive will have increased in size to give you plenty of extra room.


----------



## STICKMAN (Sep 13, 2008)

My only comment would be don't go cheap!!! Why not protect your invest ment with a decent raid system.  I would be highly upset to set up a system and have it crash on ya due to be a cheaper setup......


----------



## Moglex (Sep 13, 2008)

Remember the disadvantages of RAID.

1) You need three parallel drives to even approach the resiliance of a G/F/S system.

2) All your data is in physically the same place so theft/fire/mechanical damage are not protected against.

3) A catastrophic software problem (system malfunction/virus/trojan/malware) or user error will simultaneously overwrite/delete *all* the copies of your data at the same time.

4) If a disk fails you generally need to find one of exactly the same geometry which although it may not be a problem in the long term may cause a delay, especially if a drive fails after a couple of years.

I generally recommend against RAID out of the server room (except for striped arrays deployed for speed) as it seems to give end users a definite sense of false security.

There may be a little more work in a G/F/S system but the increase in protection for irreplacable photographs would seem to be well worth it.


----------



## Bifurcator (Sep 13, 2008)

Why USB? It's so slow... Considerations:


 Place the big drive internally - fast - cheapest solution. 
 a hot-swap bay - fast and portable/interchangeable - can be pricey.
 an external drive with an ethernet connection: 100BaseT, *1000BaseT*, Fast - slightly more expensive than USB or firewire.
 a firewire connectable external.
 USB 2.0 
 USB 1.x (or 2.0 protocol only) 

The transfer speeds relative to the above list look something like:


> 2000 megabits/sec. Full Duplex
> 2000 megabits/sec. Full Duplex
= 1000 megabits/sec. Full Duplex
~ 800 megabits/sec. Full Duplex &#8224;
~ 400 megabits/sec. Half Duplex
~ 12 megabits/sec. Half Duplex

&#8224; usually 800 these days but can be 400 on el-cheap-o's. some high grade units are 1600 and 3200 too but I think still pricey and not yet mainstream.

Half duplex means you can't read and write at the same time.

USB storage is good if you're sharing the drive between many different types of machines where a USB port is the lowest common denominator. It's very poor as a dedicated or semi-dedicated mass storage interface.


----------



## Alex_B (Sep 13, 2008)

isn't the idea of a raid to have no interruption in data availability in case a drive fails? Such as for vital server solutions. A raid is not the safe way for long term data storage but for 100% availability all the time IMHO. Hardly needed for personal photography databases. 

If you have a raid you still need a reliable backup. See also Moglex's comment.

So why not start with a backup system straight away?

backup -> prevent data loss
raid -> maintain high data accessibility 24/7

people often confuse the two.



And remember, if you do backups on HDDs, use different drives of different vendors of different production batches. identical drives tend to fail around the same time.


----------



## Alex_B (Sep 13, 2008)

oh, and i prefer external over internal since that way one backup can be located elsewhere (protection against fire, water, theft).


----------



## Moglex (Sep 13, 2008)

Bifurcator said:


> [*] Place the big drive internally - fast - cheapest solution.


And if the machine is damaged/catches fire or gets stolen, bye bye data.



> [*] a hot-swap bay - fast and portable/interchangeable - can be pricey.


Why does it need to be hot swap?

Cold swap bays are very cheap and perfectly usable provided you don't want to use the drives like CD's/DVD's.



> [*] an external drive with an ethernet connection: 100BaseT, *1000BaseT*, Fast - slightly more expensive than USB or firewire.
> [*] a firewire connectable external.
> [*] USB 2.0
> [*] USB 1.x (or 2.0 protocol only)



These are all really compromises.

Whether you need them depends on how you want to use the drive.

If you keep all your active data on the main drive in the machine and just archive/backup on the external you may just as well stick with USB2. It may be a little slower but you're not really going to start off a backup and sit and watch it.

It also has the advantage that you can connect it to just about anything.



> I've forgotten exactly but I think the transfer speeds relative to the above list look like:
> > 2000 megabits/sec. Full Duplex
> > 2000 megabits/sec. Full Duplex
> = 1000 megabits/sec. Full Duplex
> ...



These figures are often quoted but are effectively meaningless at least above USB1 as reading and writing of a disk is actually limited by how fast the drive can move data to and from the platter rather than the interconnection speed.




> It's very poor as a dedicated or semi-dedicated mass storage interface.


That depends entirely on what you are using it for.


----------



## Bifurcator (Sep 13, 2008)

Alex_B said:


> isn't the idea of a raid to have no interruption in data availability in case a drive fails? Such as for vital server solutions. A raid is not the safe way for long term data storage but for 100% availability all the time IMHO. Hardly needed for personal photography databases.



Depends on the kind of RAID. 


RAID 0 (striped disks) distributes data across several disks in a way which gives improved speed and full capacity, but all data on all disks will be lost if any one disk fails. Usually increases overall throughout ~75% with each additional drive.
RAID 1 (mirrored disks) uses two (possibly more) disks which each store the same data, so that data is not lost as long as one disk survives. Total capacity of the array is just the capacity of a single disk. The failure of one drive, in the event of a hardware or software malfunction, does not increase the chance of a failure nor decrease the reliability of the remaining drives (second, third, etc). No speed increase.
RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk. The less common RAID 6 can recover from the loss of two disks. Increases speed ~ 70% with each additional storage increasing unit.

Those are the most popular ones anyway. 0+1 and 1+0 are also fairly popular too.


----------



## Alex_B (Sep 13, 2008)

i was implying raid 5 here. no one i know these days still uses level 0 or 1 anymore.

As for raid, if all drives are the same built, often soon after the first drive fails, one or two others die too. Very bad if you do not like data loss. seen it happening often.


----------



## Moglex (Sep 13, 2008)

Alex_B said:


> As for raid, if all drives are the same built, often soon after the first drive fails, one or two others die too. Very bad if you do not like data loss. seen it happening often.



This is why I immediately recommended against it for anything other than throughput increase.

It just does not give end users the protection they need for vital data.

As you say above, it's protection element is about high accessibility not medium/long term backup.


----------



## Bifurcator (Sep 13, 2008)

Alex_B said:


> i was implying raid 5 here. no one i know these days still uses level 0 or 1 anymore.



I guess you don't know many people.   No offense. Lone Wolf? 

0, 1, 0+1, and 1+0 are still like 100 to 1 more popular than any others. Most (~95%) mother boards with on-board RAID support IDE RAID0 and/or RAID1 only. I guess in an average population of 1000 users it's real close to 50 to 100 who are using 0 or 1 and only between one and three out of that 1000 who have RAID5. 

I guess somewhere around 80 or 90 percent of RAID5 users are on high dollar SCSI drives like the Raptor  or etc. and not the more reasonable sanely priced IDEs


----------



## Tolyk (Sep 14, 2008)

Just going to skip the debate on this one, but LaCie makes great external hard drives with integrated RAID abilities, or you can forgo the raid and just use 'em as backup drives. Quite a fair price on the 1TBs, good quality.


----------



## Alex_B (Sep 14, 2008)

Bifurcator said:


> I guess you don't know many people.   No offense. Lone Wolf?



Maybe I just know a different kind of people than you do


----------



## Bifurcator (Sep 14, 2008)

I dunno... I'm in with film and video producers, 3D CG animation artists, matt and comp (composite) artists, and etc. I taught CG animation & comp for 8 years and know hundreds of post production houses and boutiques. 

Read the hardware support sites tho and 99% of the questions and recommendations are about 0 and 1 IDE configurations.

The LaCie RAID unit Tolyk mentioned is RAID0 btw.


----------



## usayit (Sep 14, 2008)

One of the nice things about mirroring externals (albeit a bit minor) is that you can split the mirror and take the other drive with you in a pinch.

Also... don't forget that redundancy doesn't protect from corruption.

Back to the OP/Vautrin, what computer do you have?  If Mac, the cheapest would be mirroring of two firewire drives (LaCie is what I like) via Mac OS X's Disk Utility (built right in).  If possible, make sure the disks are on seperate firewire controllers.   Its not fancy but it works.  The striped set can be backed via Carbon Copy Cloner to another external on a nightly basis on a automatic schedule.  

Software mirroring is also possible in windows.  If you spend a little more, you can also consider getting hardware based RAID controllers.  

I have my two SATA drives stripped for better performance.  This is my workspace, applications, O/S, and swap.  In addition, I have two externals (firewire 800 LaCie) that are mirrored for short term data redundancy.  Everything eventually gets backed incrementally to magnetic tape and/or archival gold DVDs.  

I also have a few externals from Acomdata.  They don't get good reviews but mine haven't failed yet.  They are significantly cheaper than LaCie.


----------



## Dao (Sep 14, 2008)

I am working in an environment that have over 2000 servers.  And RAID1, 5, 6, 1+0, 5+0 are common.  RAID 6 still offer redundancy even if 1 of the drive in the system fails.  Or you can install Solaris with ZFS and have the same thing with RAID Z.  I believe FreeBSD and OS X from Apple also port ZFS to its OS. I think ZFS is really the filesystem of the future.

And I do agree that the above technologies will not stop data lost.  Data backup is still needed.

If you really want to go further to protect your data (or images), Distributed File System (DFS) can help.  Theorically, you can save a image to a file system.  The system will put that file in a storage (file server) close to you.  And in the background, it will replicate the file to another file server via the LAN or WAN. The user do not know what happen.  All he/she knows is a file is save in the filesystem, the system itself will do the rest. You can have 3 servers like one in USA, one in UK and one in Japan.  Let say you save all you files in a filesystem under your folder called myimages.  Technically, all 3 servers around the world will have a copy.  Now you are in USA and you connect to the file system.  Just happen at the time you connect to the file system, the USA server is missing (i.e. due to Hurricane Ike, the Houston site is off line).  You still have access to all your files in myimages folder and do not even know one server is missing in action.

And I do believe there is a service out there offer storage space for customer to store their images.  You paid a fee and install a program in your PC.  That software will monitor your image folder in your PC for changes.  And it will replicate the files to their online storage site in the background.


Anyway, for the original poster's question, a CHEAP solution is get a external HD and move the files over and burn a copy to either CD blanks (last longer) or DVD blanks (more capacity) or even BlueRay blanks ($$).


----------



## Tolyk (Sep 14, 2008)

Bifurcator said:


> I dunno... I'm in with film and video producers, 3D CG animation artists, matt and comp (composite) artists, and etc. I taught CG animation & comp for 8 years and know hundreds of post production houses and boutiques.
> 
> Read the hardware support sites tho and 99% of the questions and recommendations are about 0 and 1 IDE configurations.
> 
> The LaCie RAID unit Tolyk mentioned is RAID0 btw.


Actually, most of the units are capable of switching from RAID0 to 1 or even 5. The LaCie rep I've spoken with thinks that RAID1 is what the majority of photographers would want from a raid system.


----------



## Moglex (Sep 15, 2008)

Tolyk said:


> The LaCie rep I've spoken with thinks that RAID1 is what the majority of photographers would want from a raid system.



Which is pretty daft for all the reasons already mentioned.

A photographer typically needs all the speed s/he can get from a disk and long term backup.

All they really get from RAID1 is a possible very small increase in read speed, a definite decrease in write speed and high availability.

A decent g/f/s external disk system will give them rock solid backup and still manage very high availability, since if the main drive goes down they can just plug in the latest backup and carry on.


----------



## Moglex (Sep 15, 2008)

Bifurcator said:


> Then just backup to CD or DVD as others have said.


CD and DVD are not suitable for backup of irreplaceable material. They are simply too unreliable. Commercially pressed CD's and DVD's may last for decades but those burned on a computer cannot be relied upon.

You can burn them, check them and find the perfect, and then at any time find they'll refuse to load.

Some people will tell you they've never had a problem but many more will have experienced failures. Do you want that sort of risk with material that can never be replaced?

The only way to be certain of retaining data is to write it to several hard disk drives and check these regularly and replace them every 2-4 years.

It may seem an expensive option but external hard drives are cheap as chips nowadays. Only you know how important your data is to you.



> If you want to be super safe use two drives: one 1TB drive and compress backup to a second drive of 320GB which you normally keep off-line.


This is nonsensical. The formats in which photographic material is stored does not compress well (if at all). Use disks of the same size (as the area on which you store your shots) for backup.

Using a single disk for backup is also a recipe for disaster. You might find that your prime drive fails and the backup chooses the next startup to konk out.

Always use *at least* two drives for backup and make sure one is physically separate from both the other and your computer.


----------



## Bifurcator (Sep 15, 2008)

Tolyk said:


> Actually, most of the units are capable of switching from RAID0 to 1 or even 5. The LaCie rep I've spoken with thinks that RAID1 is what the majority of photographers would want from a raid system.



Hehehe, he's a silly guy.  RAID1 would be for something like Video on demand.  Where you have a few hundred thousand videos across many drives (all fast enough to serve the videos). If one drive dies you pull it and replace it and the customer never knows the difference. With RAID1 you're only using 50% of your total drive space. Four 1TB HDDs in a RAID1 is 2TB with zero speed increase. Also I would be willing to bet their RAIDs can do 0 and 1 but NOT 5.

If you have four 1TB drives in a RAID0 configuration you hace 4TB of storage space with about 310% speed of a single drive.

With RAID5 three 1TB drives will give you 2TB about 160% speed of a single drive. With a RAID 5 if a drive fails you usually have to shut down to replace the bad one. 


*RAID 0*
Striped set without parity/[Non-Redundant Array]. Provides improved performance and additional storage but no fault tolerance. Any disk failure destroys the array, which becomes more likely with more disks in the array. A single disk failure destroys the entire array because when data is written to a RAID 0 drive, the data is broken into fragments. The number of fragments is dictated by the number of disks in the array. The fragments are written to their respective disks simultaneously on the same sector. This allows smaller sections of the entire chunk of data to be read off the drive in parallel, giving this type of arrangement huge bandwidth. RAID 0 does not implement error checking so any error is unrecoverable. More disks in the array means higher bandwidth, but greater risk of data loss. 
Minimum number of drives needed: 2	
*RAID 1*
Mirrored set without parity. Provides fault tolerance from disk errors and failure of all but one of the drives. Increased read performance occurs when using a multi-threaded operating system that supports split seeks, very small performance reduction when writing. Array continues to operate so long as at least one drive is functioning. Using RAID 1 with a separate controller for each disk is sometimes called duplexing. 
Minimum number of drives needed: 2	
*RAID 2*
Redundancy through Hamming code. Disks are synchronised and striped in very small stripes, often in single bytes/words. Hamming codes error correction is calculated across corresponding bits on disks, and is stored on multiple parity disks. 
Minimum number of drives needed: 3 
*RAID 3*
Striped set with dedicated parity/Bit interleaved parity. This mechanism provides an improved performance and fault tolerance similar to RAID 5, but with a dedicated parity disk rather than rotated parity stripes. The single parity disk is a bottle-neck for writing since every write requires updating the parity data. One minor benefit is the dedicated parity disk allows the parity drive to fail and operation will continue without parity or performance penalty. 
Minimum number of drives needed: 3 
*RAID 4*
Block level parity. Identical to RAID 3, but does block-level striping instead of byte-level striping. In this setup, files can be distributed between multiple disks. Each disk operates independently which allows I/O requests to be performed in parallel, though data transfer speeds can suffer due to the type of parity. The error detection is achieved through dedicated parity and is stored in a separate, single disk unit. 
Minimum number of drives needed: 3 
*RAID 5*
Striped set with distributed parity. Distributed parity requires all drives but one to be present to operate; drive failure requires replacement, but the array is not destroyed by a single drive failure. Upon drive failure, any subsequent reads can be calculated from the distributed parity such that the drive failure is masked from the end user. The array will have data loss in the event of a second drive failure and is vulnerable until the data that was on the failed drive is rebuilt onto a replacement drive.
Minimum number of drives needed: 3 
*RAID 6*
Striped set with dual distributed Parity. Provides fault tolerance from two drive failures; array continues to operate with up to two failed drives. This makes larger RAID groups more practical, especially for high availability systems. This becomes increasingly important because large-capacity drives lengthen the time needed to recover from the failure of a single drive. Single parity RAID levels are vulnerable to data loss until the failed drive is rebuilt: the larger the drive, the longer the rebuild will take. Dual parity gives time to rebuild the array without the data being at risk if one drive, but no more, fails before the rebuild is complete. 
Minimum number of drives needed: 4

And there's some other weird combo configurations too like RAID15.  RAID0 is what you want for large fast storage. Then just backup to CD or DVD as others have said. But actually I think with 1TB HDDs being pretty common now no one needs RAID for photographs. If you want to be super safe use two drives: one 1TB drive and compress backup to a second drive of 320GB which you normally keep off-line.


----------



## Moglex (Sep 15, 2008)

Bifurcator said:


> Then just backup to CD or DVD as others have said.


CD and DVD are not suitable for backup of irreplaceable material. They are simply too unreliable. Commercially pressed CD's and DVD's may last for decades but those burned on a computer cannot be relied upon.

You can burn them, check them and find them perfect, and then at any time find they'll refuse to load.

Some people will tell you they've never had a problem but many more will have experienced failures. Do you want that sort of risk with material that can never be replaced?

The only way to be certain of retaining data is to write it to several hard disk drives and check these regularly and replace them every 2-4 years.

It may seem an expensive option but external hard drives are cheap as chips nowadays. Only you know how important your data is to you.



> If you want to be super safe use two drives: one 1TB drive and compress backup to a second drive of 320GB which you normally keep off-line.


This is nonsensical. The formats in which photographic material is stored does not compress well (if at all). Use disks of the same size (as the area on which you store your shots) for backup.

Using a single disk for backup is also a recipe for disaster. You might find that your prime drive fails and the backup chooses the next startup to konk out.

Always use *at least* two drives for backup and make sure one if physically separate from both the other and your computer.


----------



## usayit (Sep 15, 2008)

I think you guys have gone off the deep end.  A few points from the OP:

1) He never mentioned anything about business critical machines
2) His term was "computer's".  Meaning 1 desktop.  Not a server farm.  We are not event talking about a single server.
3) He said CHEAP.  We are talking best buy cheap.  We are not talking about EMC frames, hot swap storage units, etc...
4) His term was "USB".  At this point, it is pointless talking about high performance RAID 5.  

I'm in the storage and backup/recovery business.  Even with that experience, what I do at home is completely different as the purpose and final goal is different.  The worst thing to do while communicating with the customer is not listening.  I bet this thread is utterly useless to the OP and is now just a contest.


----------



## usayit (Sep 15, 2008)

Moglex said:


> This is nonsensical. The formats in which photographic material is stored does not compress well (if at all). Use disks of the same size (as the area on which you store your shots) for backup.



Not true.  Uncompressed Tiff just to name one.


----------



## usayit (Sep 15, 2008)

Vautrin (if you are still following this thread).  If Mac, read my previous post.  If windows, I suggest getting a Firewire card and running simple RAID 1.  Here is a link for you:

http://eshop.macsales.com/shop/firewire/

Yes I know it says "Mac" in their domain name but all their firewire external hard drive products should be compatible with Windows machines assuming you install a firewire card.  They are easy to deal with.


----------



## Moglex (Sep 15, 2008)

usayit said:


> Not true.  Uncompressed Tiff just to name one.


Just because there may be formats which compress does not mean it's generally true that they do.

RAW and JPG certainly don't.

If you are certain that you will continue to use something such as uncompressed TIFF, by all means audit your storage and calculate what percentage of your files will compress, then calculate by how much on average, then calculate a safety margin and finally save a small amount but in general you're probably better off just matching the storage allocated on your main drive.


----------



## Moglex (Sep 15, 2008)

usayit said:


> I think you guys have gone off the deep end.  A few points from the OP:
> 
> 1) He never mentioned anything about business critical machines
> 2) His term was "computer's".  Meaning 1 desktop.  Not a server farm.  We are not event talking about a single server.
> ...



*Exactly!*

That is why Alex B and myself have been playing down RAID and all its complexities and asking OP to consider what is really important in an archive/backup.

Great long screeds about RAID types really aren't helpful here.



usayit said:


> Vautrin (if you are still following this thread).  If Mac, read my previous post.  If windows, I suggest getting a Firewire card and running simple RAID 1.



'A' simple raid 1?

That's fine if he's prepared to lose any archived data that resides on the RAID. It doesn't really address backup considerations. Setting yourself up with huge RAID systems without *very carefully* considering backup is foolhardy to say the least.

As to firewire, by all means go for that if you are certain that every machine you may want to connect the array to has a firewire port.

Be aware, though, that extreme high speeds actually aren't that important for archive and backup and at 50MB/s USB2 really isn't sluggish. It may be instructional to get a disk benchmark program and find out how fast your machine actually does read and write data!


----------



## Dao (Sep 15, 2008)

hahaha .. it seems that the OP maybe scared to come back because of the direction of this thread is heading.

Anyway, As I said, the cheap solution is just move the files to a external harddisk. And make an backup with a quality DVDs like Taiyo Yuden.  If you really think the shelf life of 30 years of those quality DVDs is not enough, back it up on another cheap harddisk and store it somewhere.


----------



## JerryPH (Sep 15, 2008)

Alex_B said:


> i was implying raid 5 here. no one i know these days still uses level 0 or 1 anymore.
> 
> As for raid, if all drives are the same built, often soon after the first drive fails, one or two others die too. Very bad if you do not like data loss. seen it happening often.



Actually, if you want maximum speed AND maximum fault tolerance *both* RAID 1 and 0 in tandem are what you want (RAID10).

RAID 5 is reserved for servers, rarely for workstations, but it is done.

For photography use, I suggest people look at the WORLDBOOK 2TB setup (basically two 1TB hard drives), but mirror it down to 1TB total space.

It is either USB 2.0 or a 10/100/1000 network connection in an external unit.  For home use this is about as high as 99% of the people need.

Now if you want to get fancy, I have an 18TB SAN (fiber optic) off of 40 1TB SCSI drives in my basement.  Yeah, its massive overkill for my needs... but it was also free.


----------



## usayit (Sep 15, 2008)

Moglex said:


> *Exactly!*
> 'A' simple raid 1?
> 
> That's fine if he's prepared to lose any archived data that resides on the RAID. It doesn't really address backup considerations. Setting yourself up with huge RAID systems without *very carefully* considering backup is foolhardy to say the least.



No kidding?  I work for the market leader for enterprise DR.  BUT the OP didn't ask about backup strategies.  I also mentioned that redundant disks doesn't protect from data corruption.

On my Mac, setting up mirror was pretty much free and easy (all you need is the disks).  Can't be any more simple or cheap.


----------



## Vautrin (Sep 15, 2008)

LOL...

Wow, just wow.

So I'm going to go with a G/F/S system.  Although I'm thinking G/F only because no need for a third disk.  G is going to be a USB 500 GB and F is going to be a network drive if I can find one, and then hopefully I can set up a script to rsync the drives on my home network or something...

And, yes, as many people had pointed out I was looking at RAID not for speed or high end server room needs but because I want my data to be safe.  I figured RAID would be easiest to do this because if one drive died I could just pull it and replace it and have it mirror, but people have brought up good points -- especially about locational hazards.

Anyone know of a good online backup service that I could get to mirror my 500GB for cheap?

And as for Windows / Mac that is a problem -- saving for a new Mac but currently one a windows box...

-Dan


----------



## JerryPH (Sep 15, 2008)

The best online systems cost $$, and sometimes even more than if you do it yourself... plus on top of it, you will NEVER know if *they* backup their drives or not.

I prefer in trusting #1... me... lol.  It's just not that hard!


----------



## Bifurcator (Sep 15, 2008)

seconded.


----------



## Mystwalker (Sep 15, 2008)

Not sure you need RAID at all - in fact I will advise against it.

I was under impression RAID 0 will give you "best" performance - it did until something went wrong with one of the disks requiring I format both and reinstall everything.  Luckily, I'm paranoid and have double storage of everything on CD/DVD and websites.

I would not use RAID 1 nor 5 unless you are constantly touching the pictures you have stored on them.  RAID is not really an "archive" solution (imo).

Today, I prefer a USB solution such as Seagate's FreeAgent Pro.  You can get 750 to 1TB for around $120 to $180 depending on where and when you shop.  Definitely not even "quick", but it is only for archiving.

I also backup everything to CD/DVD.


----------



## usayit (Sep 15, 2008)

Mystwalker said:


> I would not use RAID 1 nor 5 unless you are constantly touching the pictures you have stored on them.  RAID is not really an "archive" solution (imo).



You are right... its not an archival solution BUT you have to think of it as two seperate things: 1) redundant disk storage 2) archival storage.

RAID 1 is so cheap that there is no reason why you should not use it for interim storage between workspace and archival space.


----------



## Bifurcator (Sep 15, 2008)

RAID1 Sucks nuggets! In a home computing environment it's ONLY for the extreme anal retentive who doesn't understand RAIDs at all. I seriously believe that. So do most of the IT old-timers I know.

When a drive breaks 90% of the time it corrupts data for a few weeks or months first. In a RAID1 configuration all of that corruption will be copied over to your other drive(s). Drives do break suddenly but it's rare and that's ALL RAID1 is good for. It's a total waste of money. You are buying twice the hardware and getting absolutely nothing for it except a higher power bill and probably more ambient room noise.

Personal Computing + RAID1 = SUCKS!


----------



## usayit (Sep 15, 2008)

Corruption (from various reasons) will impact any redundant disk setup.  The most effective protection from corruption is backup (I've mentioned this 3 times in this thread now).  In my many years in the field, I've seen tons of disks failures and rarely do they result in disk corruption.  If I had to chose priority between backup and redundancy, backup would be my number one priority.  (My O/S and workspace live on RAID 0 with nightly backup). 

Last time I checked, 1 TB disk was around $200-$300 USD.  Hardly an expensive option.   RAID 5 and 6 require 3 or more disks are usually a RAID controller.  At least on my MAC, RAID 1 is basically free  (so is RAID 0 but thats not redundant).

Again.. I think you are loosing sight of the OP.  He asked for non-enterprise/cheap/raid.


----------



## table1349 (Sep 15, 2008)

Well this has been a real pretty discussion.  Fact of the matter is, if you *are not* using an off-site secure underground disaster proof data storage company, then it is all just a moot point.


----------



## usayit (Sep 15, 2008)

Mine is... I "sneak" my stuff in with packages stored at Iron Mountain..  :thumbup:

seriously though.. thats overkill... I just do it because its "free"


----------



## Bifurcator (Sep 15, 2008)

If you're not getting drives for free it's not free. 

Why are you in love with the word "redundant" and why would you suggest to the OP to spend twice the money for half the space and then say it's the cheepest solution? 

I'm not trying to attack here but it just doesn't make any sense at all. With RAID1 there's no increased speed, there's no increased storage space, and there's no protection from read/write error (in fact with RAID1 read and write errors propagate to the mirror ). You are only protected from a single scenario where a single drive busts suddenly - which almost never happens.


----------



## usayit (Sep 15, 2008)

Bifurcator said:


> If you're not getting drives for free it's not free.



Short of the disks (which are cheap) RAID 1 built-in with MAC is free.  I did mention this earlier.



> Why are you in love with the word "redundant" and why would you suggest to the OP to spend twice the money for half the space and then say it's the cheepest solution?



Because I listen to the OP (customer).  He/She said they wanted cheap RAID.  The "R" in RAID means redundant.  It is as simple as that.  Mirrored disks simply means that a failed disk has a good chance of uninterrupted service; Disk fails, split mirror, get a new disk, re-sync the data. 

DATA PROTECTION is done at backup.  HIGH AVAILABILITY/FAULT TOLERANCE is done at mirror.

You expect me to listen and respond to you but you are not listening to what I'm posting nor what the OP posted.


----------



## usayit (Sep 15, 2008)

"Also... don't forget that redundancy doesn't protect from corruption."

"On my Mac, setting up mirror was pretty much free and easy (all you need is the disks)."

"In my many years in the field, I've seen tons of disks failures and rarely do they result in disk corruption."

"If I had to chose priority between backup and redundancy, backup would be my number one priority."


----------



## Bifurcator (Sep 15, 2008)

I'm listening. I just know better than to believe some of it. 

RIAD is RAID. RAID0 is RAID. Are you saying it's not?

"FAULT TOLERANCE" = Hardware level only!  If the hardware fault causes a data errors (which is almost always the case) you still loose.

BTW, I'm on a Mac too. Mac Pro 8-core. But I've always used RAIDS for storage. Since they were 1st available 25 or so years ago. I can't think of a time I have been RAIDless actually.


----------



## usayit (Sep 15, 2008)

Bifurcator said:


> I'm listening. I just know better than to believe some of it.



I will say the same about your U.S. conspiracy theories and leave it at that.


----------



## Bifurcator (Sep 15, 2008)

usayit said:


> "In my many years in the field, I've seen tons of disks failures and rarely do they result in disk corruption."



How do you know if the drive was broken?   In a RAID1 you'll know because all the read errors that happen while it busted will corrupt the image of the mirror. So you have an operational device that you can see is obviously trashed. 



> "If I had to chose priority between backup and redundancy, backup would be my number one priority."



Agreed. But the OP was talking about *storage* not backup nor redundancy. So nya nya. :hug::


----------



## Bifurcator (Sep 15, 2008)

usayit said:


> I will say the same about your U.S. conspiracy theories and leave it at that.



I have no conspiracy theories! Not even one.


----------



## Dao (Sep 15, 2008)

usayit. I agree with you. It is common that a lot of people misunderstand RAID technology. I'd been working in Data Center environment for more than 8 years, and I always come across with clients or prospects think RAID technology will give them reliability so that they do not need backup.  The fact and the matter is, it only give them an option that they do not need to shutdown the server while it is in a recovery mode or fix it later (off peak hours).  Will RAID increase the Reliability of the drive system?  Yes it will, see this.  (of course, I am not talking about RAID 0) However, by no mean it can prevent data loss.

As for the RAID 1, it is still very popular.  Simple and works.  If someone is not using expensive hardware RAID card, RAID 1 is the best approach. Software RAID or peusdo HW RAID (like most of the RAID that came with motherboard) is better to use RAID 1 unless you do not care about speed.

Also, if the system crash and you need to recover data from the dirve/drives, it is easier to do it with RAID 1.


----------



## Bifurcator (Sep 16, 2008)

Dao said:


> better to use RAID 1 unless you do not care about speed.



You mean unless you DO care about speed tho right?  RAID1 is zero speed increase.

http://www.anandtech.com/storage/showdoc.aspx?i=2101&p=3 <-- read'em and weep.


----------



## Dao (Sep 16, 2008)

Bifurcator said:


> You mean unless you DO care about speed tho right?  RAID1 is zero speed increase.



I do mean "DO NOT" since Software RAID 5 (or RAID 6 if possible) will use too much CPU processing power, so the degrade in overall performance of RAID 1 is a lot less than RAID 5 with software RAID especially in rebuild mode.

If you use a true hardware RAID card that has a build-in processor to handle the RAID function is a different story.

If someone would like to increase the speed of a RAID set, they will usually stripe 2 of them to create a RAID 0 on top of the current RAID sets.  i.e. RAID 1+0, or 5+0


----------



## Bifurcator (Sep 16, 2008)

Yeah, that's kinda true for chipset-only RAID5. But since you mentioned RAID0 just before that I thought you were considering it as well. RAID0 scales the best while at the same time maximizing the storage space to number of units ratio. And it consumes no more CPU usage than RAID1 which of course you must know does not increase read or write speed (performance) at all.

1+0 is OK but kind of a waste of money and electricity. I at least contend that for a home user full unit mirroring  - redundant parallelism - is about as useful as tits on a bull - they might be nice to have but the advantages are just about nil in the real world. 

I can see periodic backups to something like a USB or Firewire drive tho. 

Anyway I thought you were saying: "better to use RAID 1 [than RAID 0] unless you do not care about [file access and I/O] speed." which is of course false and what usayit and I were discussing.


----------



## Moglex (Sep 16, 2008)

Bifurcator said:


> When a drive breaks 90% of the time it corrupts data for a few weeks or months first. *In a RAID1 configuration all of that corruption will be copied over to your other drive(s).*



You really do talk some unmitigated nonsense, don't you?

There is no path from one disk in a RAID array to another short of reading the data into the computer at which point any corrupt data will be identified.

So one drive going faulty will *NOT* cause corrupt data to be written over other drives.

OTOH, you are correct in saying that RAID1 is all but useless for the home user.


----------



## Moglex (Sep 16, 2008)

Bifurcator said:


> How do you know if the drive was broken?   In a RAID1 you'll know because all the read errors that happen while it busted will corrupt the image of the mirror. So you have an operational device that you can see is obviously trashed.



More complete nonsense, I'm afraid.

You seem to be claiming some sort of expertise in this field but you also do not appear to have a clue about the general systems architecture of even basic disk technology.

In case anyone has been misled by what you've written, a few basic facts:

1) Before a packet of data is sent to a disk the OS calculates some check data from the data to be sent. This is written with the user data.

2) The drive itself calculates check data that is written with the data sent from the OS.

3) The drive *checks* the data to make sure it has written the data correctly.

4) Upon reading the data, the drive checks that the check data tallys with the user data - if it doesn't it raises an error.

5) The RAID system (if in use) will check that both drives have returned the same data.

6) The OS will make its own check to see that the check data matches the user data retrieved.


So having one faulty drive will *not* corrupt the entire array. The array will not be allowed to function if its integrity cannot be assured.

We are agreed that RAID1 is pointless for end users (unless they have some *very* unusual application), but it is nowhere near as useless as you make out when used in the appropriate environment.


----------



## Moglex (Sep 16, 2008)

Bifurcator said:


> I can see periodic backups to something like a USB or Firewire drive tho.



But you have already indicated that you believe that a faulty drive can go on writing bad data to a disk without detection even in a RAID1 array.

So what on earth would be the point of backing up that data (in your world)?


Of course, in actual fact, although a faulty drive can write bad data to a disk without flagging an error, as soon as that bad data is accessed the fault is discovered which is why:

1) RAID1 arrays do work.
2) You don't corrupt your backups with corrupt data from a faulty disk.


----------



## JerryPH (Sep 16, 2008)

Bifurcator said:


> RAID1 Sucks nuggets! In a home computing environment it's ONLY for the extreme anal retentive who doesn't understand RAIDs at all. I seriously believe that. So do most of the IT old-timers I know.
> 
> When a drive breaks 90% of the time it corrupts data for a few weeks or months first. In a RAID1 configuration all of that corruption will be copied over to your other drive(s). Drives do break suddenly but it's rare and that's ALL RAID1 is good for. It's a total waste of money. You are buying twice the hardware and getting absolutely nothing for it except a higher power bill and probably more ambient room noise.
> 
> Personal Computing + RAID1 = SUCKS!



I'm not sure that you understand how RAID1 (mirroring), works based on what you said there.

In a software RAID1 configuration (lets say XP/Vista/Server scenario) if the primary drive dies, your machine doesn't boot.  In a hardware RAID1 scenario, depending on the RAID hardware manufacturer, it may just beep, poke you in the shoulder, send emails, scream at you that one drive is damaged and keep the system up and running by automatically switching over to the "good" drive.  In either case, if the secondary drive dies, you are notified one way or another (beeping, popup message on screen, etc...).

In a RAID setup, information is NOT continually copied from one drive to another any other time other than during the creation of the mirror (unless you happen to have some 1970's ESDI RAID1 controller?).  

If info on 1 drive is corrupt BEFORE the mirror, and you create a mirror, THEN and only then is corrupted data written to the mirror.  All other times, data in the cache (RAM) is written first to the primary drive and then that same data from RAM is written to the secondary drive (in an IDE or SATA setup) or written concurrently simultaneously in a SCSI setup.

Honestly, for a home user, a RAID1 external solution is probably the best bang for the buck.  It means that they have 2 copies of whatever they copy to their external array.  It also means that they don't have to purchase a unit with more than 2 drives and for archival purposes, do not need BLAZING speed to get the data there... just turn on the drive(s), copy the data there on a regular basis (once a week perhaps if they take that many pictures?) and turn them off right afterwards, even further increasing the lifespan (MTBF) of the hard drives.  

Doing it this way means you are spending a lot less and accomplishing your goals... which is to find an effective and economical means of protecting your data.

Of course, to date, there are no realistic 100% effective solutions that last forever, so it doesn't hurt to hedge your bets by diversifying media.  I use a SAN and a couple of 1 and 2 TB external drives, but I also save on my local hard drives (my local HDs on most of my computers are RAID0 of 3-4 drives depending on the computer... zero redundancy, but great for speed), and DVDs.  It all depends on how much you value your data and how much you want to spend.  One cannot get it much better in terms of least $ spent and effectiveness than an external 2 drive RAID1 setup.


----------



## Moglex (Sep 16, 2008)

JerryPH said:


> I'm not sure that you understand how RAID1 (mirroring), works based on what you said there.
> 
> In a software RAID1 configuration (lets say XP/Vista/Server scenario) if the primary drive dies, your machine doesn't boot.  In a hardware RAID1 scenario, depending on the RAID hardware manufacturer, it may just beep, poke you in the shoulder, send emails, scream at you that one drive is damaged and keep the system up and running by automatically switching over to the "good" drive.  In either case, if the secondary drive dies, you are notified one way or another (beeping, popup message on screen, etc...).
> 
> ...



Thanks for backing up my assertion that some of the posts above contain complete nonsense.



> Honestly, for a home user, a RAID1 external solution is probably the best bang for the buck.  It means that they have 2 copies of whatever they copy to their external array.  It also means that they don't have to purchase a unit with more than 2 drives and for archival purposes, do not need BLAZING speed to get the data there... just turn on the drive(s), copy the data there on a regular basis (once a week perhaps if they take that many pictures?) and turn them off right afterwards, even further increasing the lifespan (MTBF) of the hard drives.
> 
> Doing it this way means you are spending a lot less and accomplishing your goals... which is to find an effective and economical means of protecting your data.
> 
> ...



I don't want to end up going around in circles but I still cannot see how a RAID1 setup can possibly be better than backing up to a cycle of 2 or more external hard drives.

I have a system whereby all new or changed data on the target drives is written to the external drive. This *dramatically* speeds up the process compared with just writing everything out (of course, you can do that to a RAID array as well). I have three drives which are used in turn with one always being kept physically separate.

The advantage of this scheme is that even if my system were to be stolen, burned beyond repair in a fire, or otherwise fatally damaged I would still have all my vital data. (Recent data is kept on a web based store until it has migrated to all backup drives).

The only advantage I can see to a RAID solution is that you don't need to keep track of which backup to use or move the off-site one around but these tasks seem very minor given the vastly greater security achieved.


----------



## Dao (Sep 16, 2008)

I think it all comes down to if the cost for the RAID 1 solution worth the money.  And it also varies from person to person.

Someone can get 2 or more of those RAID1 USB/eSATA storages and feel happy about it.  However, some others may be tight in budget and can only afford one USB drive.  Either case, it's still a solution.

Off Topic:
For those who like to play around with technology, have some older systems laying around and you as well as someone you can trust that have a decent speed broadband connection, you may want to take a look at OpenAFS.  You maybe able to achieve automatic files repulications between 2 sites with the distributed Network Filesystem.


----------



## Dweller (Sep 16, 2008)

Vautrin said:


> Anyone know of a good online backup service that I could get to mirror my 500GB for cheap?



A thread on another forum I was reading this morning was discussing Carbonite, and people seem to really like it. I understand there are coupon codes that can be found online as well. They claim unlimited storage is available. I am looking at something like this myself since my current backup system still has everything in the same location.

PC --> server with data mirroring (Windows Home Server) --> tape drive --> (something offsite, likely Carbonite or something similar)

I have never lost a single photo despite some hardware loss along the way. I have dodged that bullet once, and I don't want to have to worry about it anymore.


----------



## Dweller (Sep 16, 2008)

Dao said:


> Off Topic:
> For those who like to play around with technology, have some older systems laying around and you as well as someone you can trust that have a decent speed broadband connection, you may want to take a look at OpenAFS.  You maybe able to achieve automatic files repulications between 2 sites with the distributed Network Filesystem.



Wow, thanks  I was talking about something like this here a while back. I am not surprised to see a solution exists.


----------



## usayit (Sep 16, 2008)

Moglex said:


> I don't want to end up going around in circles but I still cannot see how a RAID1 setup can possibly be better than backing up to a cycle of 2 or more external hard drives.



Backup is data protection.
Mirror is fault tolerance. 

Backup scheme only doesn't provide the convenience of uninterrupted service.  Once a drive fails, you have to restore the data from your latest image.

Mirror only scheme only provides fault tolerance.  Once a drive fails, you have the convenience of continuing with  your work.  Replace the failed drive ASAP.  If corruption (doesn't happen often) or both drives fail, you are out of luck.  Data loss.. game over.

For a backup and mirror scheme, leverages both.  Once a drive fails, you have the convenience of continued service from your machine.  If a both drives fail or corruption, you have restores available from backup images.  No interrupted service, no/limited data loss.

Place an importance on Backup.  Mirror is the first line of defense against a disk failure.  If data is important but RTO is not, implore backup scheme.  If data is not important but RTO is, implore fault tolerance scheme.  If data and RTO is important, implore both.  If data and RTO is VERY VERY IMPORTANT and very valuable, purchase full RAID 5 or 6 with enterprise data protection.

RAID 1 IS the most simplistic and best bang for the buck for redundant disks... RAID 5 and 6 are of course superior but at a cost of additional hardware (software RAID 5 or 6 is highly CPU and resource heavy).  People who say RAID 1 is expensive need to look at the hardware requirements for implementing full RAID 5-6.  RAID 1 is no where the cost of RAID 5.


----------



## Moglex (Sep 16, 2008)

usayit said:


> Backup is data protection.
> Mirror is fault tolerance.
> 
> Backup scheme only doesn't provide the convenience of uninterrupted service.  Once a drive fails, you have to restore the data from your latest image.


Correct. But the OP wanted a cheap scheme and as an end user is very unlikely to want to double the cost of that solution to get high availability.



> Mirror only scheme only provides fault tolerance.  Once a drive fails, you have the convenience of continuing with  your work.  Replace the failed drive ASAP.  If corruption (doesn't happen often) or both drives fail, you are out of luck.  Data loss.. game over.


Correct.



> For a backup and mirror scheme, leverages both.  Once a drive fails, you have the convenience of continued service from your machine.  If a both drives fail or corruption, you have restores available from backup images.  No interrupted service, no/limited data loss.



Correct, but the OP said he wanted a cheap solution.

Anyway, this does not answer my objection.



> Honestly, for a home user, a RAID1 external solution is probably the best bang for the buck. It means that they have 2 copies of whatever they copy to their external array. It also means that they don't have to purchase a unit with more than 2 drives and for archival purposes, do not need BLAZING speed to get the data there... just turn on the drive(s), copy the data there on a regular basis (once a week perhaps if they take that many pictures?) and turn them off right afterwards, even further increasing the lifespan (MTBF) of the hard drives.
> 
> Doing it this way means you are spending a lot less and accomplishing your goals... *which is to find an effective and economical means of protecting your data*.
> 
> ...



You have not explained how RAID1 can possibly be better at protecting your data when it only handles failure of a drive and does nothing to protect you against theft, fire or other physical damage whilst external backup protects against all four eventualities.

The above quote strikes me as entirely wrong. You certainly can get *much* better protection for your data by getting a couple of cheap USB disk caddies. You protect against the whole gamut of dangers mentioned for about £20 more than the cost of the drives.


----------



## JerryPH (Sep 16, 2008)

Moglex said:


> You have not explained how RAID1 can possibly be better at protecting your data when it only handles failure of a drive and does nothing to protect you against theft, fire or other physical damage whilst external backup protects against all four eventualities.
> 
> The above quote strikes me as entirely wrong. You certainly can get *much* better protection for your data by getting a couple of cheap USB disk caddies. You protect against the whole gamut of dangers mentioned for about £20 more than the cost of the drives.





			
				JerryPH said:
			
		

> Honestly, for a home user, a RAID1 *external* solution is probably the best bang for the buck.



I did not think it needed explaining. It handles fire and theft in the same identical manner your USB caddy would.  It is external, it is portable (put it in your fire-proof safe or external location after backing things up if this is important to you, it *is* superior to a single drive.  The cost is marginally higher, yes... but a USB caddy is a single drive and smaller case.  Well worth the minor cost difference vs the benefits.

I think we'ved jaw-wagged this topic to death... lol


----------



## Moglex (Sep 16, 2008)

JerryPH said:


> I did not think it needed explaining. It handles fire and theft in the same identical manner your USB caddy would.  It is external, it is portable (put it in your fire-proof safe or external location after backing things up if this is important to you, it *is* superior to a single drive.  The cost is marginally higher, yes... but a USB caddy is a single drive and smaller case.  Well worth the minor cost difference vs the benefits.



It may be slightly superior to a *single* drive but for the same money you can put the drives in separate external caddies which gives you the advantage that you can have one next to the machine ready to backup and one physically secured.

So you don't skip backups because your one backup box is elsewhere and yet you always have one copy secured.

That just strikes me as a much better bang for your buck than spending it on something (i.e. the extra disk in a RAID1 array) that is designed for high availability and not long or medium term backup.



> I think we'ved jaw-wagged this topic to death... lol


Not sure about that as people still seem to think that RAID1 is a backup solution (or at least superior in some way to using the same number of disks independantly in a backup solution).


----------



## Dao (Sep 16, 2008)

RAID 1, simple, easy, why not.

think about this way.  If you have a hotswapple drive bay.  You can break the mirror and then take one of the drive to work and put another one in the back to rebuild the raid without using any additional cloning software.  Now you have a off site backup. (I am not talking about system boot drive here)  Just a thought.


As for the  single USB drive vs single USB RAID1 drives. Both are fine solutions 

It just depends on how much a person willing to spend.  If he/she can afford it, why not go with the RAID1.  It is like Canon XSI vs 40D.


----------



## usayit (Sep 16, 2008)

Moglex said:


> Anyway, this does not answer my objection.



OP's title = "Best (Cheap) RAID / Storage Solution?"

The only thing cheaper than RAID 1 is no RAID at all (which doesn't fit the OP's criteria)....  Whether or not it meets your idea of an ideal home environment is irrelevant.  OP already specified two things; Cheap and RAID.

This thread reminds me of those "Help me make a DSLR choice" threads.  They start out like, "I want a CANON DSLR but I can't make up my mind".  Then some Nikon shooter chimes in and says "you want a Nikon [insert model here]".  Then the thread degrades into a Canon versus Nikon debate which is UTTERLY and COMPLETELY useless to the OP since they already specified in the starting post that they wanted CANON.


----------



## Moglex (Sep 16, 2008)

usayit said:


> OP's title = "Best (Cheap) RAID / Storage Solution?"
> 
> The only thing cheaper than RAID 1 is no RAID at all (which doesn't fit the OP's criteria)....  Whether or not it meets your idea of an ideal home environment is irrelevant.  OP already specified two things; Cheap and RAID.



Indeed that is what he specified but if it appears that someone may have made a false assumption about something (in this case the desirabilty of RAAID) it is as well to present the alternatives.

OP has now said that he intends to go with a g/f backup approach so in this case suggesting the alternative with reasons proved useful. Had he said that he wanted to continue with RAID it would have been a different matter.



> This thread reminds me of those "Help me make a DSLR choice" threads.  They start out like, "I want a CANON DSLR but I can't make up my mind".  Then some Nikon shooter chimes in and says "you want a Nikon [insert model here]".  Then the thread degrades into a Canon versus Nikon debate which is UTTERLY and COMPLETELY useless to the OP since they already specified in the starting post that they wanted CANON.



Except:

1) If you say you know you want a Canon it is likely that you have some reason for pre-empting the choice of manufacturer. Any end user sayng they want RAID without specifying why is very likely to have been misled about the advantages of RAID.

2) In this case, as already mentioned, OP has looked at the information given and decided to go with the more robust solution of two independant external drives so in no way was providing alternative infomation "UTTERLY and COMPLETELY useless to the OP".


----------

