"After some delving into the problem, we found that what appeared to be URE's were in fact time-outs - the RAID controller was being kept so busy writing new data as well as handling the rebuild, that it was unable to deal with bad sectors in the normal way (moving blocks around) and so they 'timed-out' and were flagged as unrecoverable. However if we took the RAID offline and executed a check/repair routine, most of those URE's were resolved."
Barry, that's great advice. I've seen the same kind of thing happen on a variety of units (particularly cheaper pro-sumer models), and the removed drives invariably pass every test thrown at them... this explains it! Seeing as my usual design for systems incorporating RAID involves all data normally being written to the RAID, and an internal drive or two in the DVR for backup when the RAID is offline, this is a perfectly workable solution.
Now to the original questions:
"The first obvious one that strikes me is 'having too many eggs in one basket' - ie if a 4TB (non-RAID) drive fails (yes even Enterprise drives can fail!) the amount of video evidence lost is significantly more."
Funny thing is, I've seen the same discussion every time another major leap is made in storage capacity. "Should I use the new 1TB drives, or stick with a bunch of 250GB?" and so on. Before long, we'll be debating whether to incorporate newly-available 10TB drives, or just go with good ol' reliable 4TB models...
"RAID overhead. The rule of thumb that in a RAID-5 config you lose the capacity of one drive (plus a bit more) means that 32TB, configured as 8 x 4TB HDDs, will yield roughly 28TB at RAID-5, whereas the same config using 16 x 2TB drives will give you 30TB useable."
In addition to Carl's points about "diminishing returns" on drive populations, consider the cost difference: at a local retailer, a Western Digital 4TB RE "Enterprise" drive lists at about $430, while a 2TB RE is $221. Cost for eight 4TBs = $3440, cost for 16 2TB = $3536, so despite being a fraction less space (about, what, 7-8% less?), it's also cheaper. Factor in things like less heat generated, less power consumed... the fact that a 16-bay RAID is going to cost substantially more than an 8-bay model (and probably take up more rack space as well)...
Optionally, the 3TB RE runs about $320, and 11 of those in RAID5 would give you the same 30TB while landing right around Carl's "sweet spot", for $3520 :)
Of course, if you're looking to go RAID6 or RAID6+DS, you have to re-calculate everything... :)