"For example, you will agree I assume that some VMSes use SQLServer as their underlying storage mechanism. "
You are correct, Rukmini, I also forgot to take that into account. I'm just not used to working with systems like that. The systems we work with either use their own proprietary database management layer where you can span volumes and it still works fine if you loose a volume, or it uses a flat file system to store chunck of video segements into folder and file structures, and when you query recordings all it does is scan the designated storage space looking up the corresponding folder and file names.
If the VMS system uses something like SQL or other database system that was not designed for video surveillance and is not tolerant of segments going missing, then yes you really have problems.
Thank you Rukmini, Carl and others for helping keep me honest!
"Undisclosed, Splitting a 16-bay chassis into two 6+2 RAID groups should yield 36TB with 3TB drives"
You're right, I screwed that math up.
Splitting a 16-bay chassis into two 6+2 RAID groups should yield 36TB with 3TB drives, assuming no hot spare. With one global hot spare, yield would be 33TB.
There are also other options. We have two systems holding long term evidence that utilize a 12-bay chassis and a 12-bay JBOD split into 10+2 and 9+2 RAID groups with one shared global hot spare. These are populated with 2TB drives so the yield is 38TB, less Windows overhead.
Something to look into would be the Veracity Coldstore, does not work with all VMS but a neat solution. http://www.veracityglobal.com/products/storage-for-video-surveillance/coldstore.aspx
That's correct. In fact, the larger the hard disks, the less drives should be in any RAID group. I agree there is a point of diminishing returns, where RAID groups become so small that more disks are dedicated to parity than to actual data storage. At that point, traditional RAID will have become way too inefficient. Although I haven't had much time to thoroughly research alternatives, I understand that there are a few out there, including Drobo's BeyondRAID, Lime Technology's UnRAID, Level 1 storage redundancy and various forms of storage clustering.
I have been approached by a few companies that offer non-traditional storage recently but had other projects in work so I've begged off. I intend to dig deeper as time permits.
The reason I suggested JBOD with assigned cameras to drives was to insure your statement of
If you loose one drive in your non-RAIDed enclosure, you've only lost 3TB, or 7% of your video.
was correct, in other scenarios I'm not sure.
For example, you will agree I assume that some VMSes use SQLServer as their underlying storage mechanism. Similar to RAID you are then sub-contracting your volume management out to a third party. When SQL creates a tablespace that spans drives and then you lose a drive without redundancy or backup, thereby losing numerous chunks containing partial records, btree indices pointing to other drives and i-noded blob extensions and the like, what makes you think that your whole database isn't going to look like confetti? Same goes with any other VMS managed dataspace allocation scheme that might stripe or keep indexes on separate drives or anything that could create multivolume dependencies.
I'm not saying that there aren't VMSes that would work as you say, but I think you would need to see how the spanning was implemented, and if such capabilities are not explicitly called out by the mfr spec, you might want to do your own test.
Rukmini, in most VMS systems you merely assign storage volumes (freely available disk space) and the VMS system spans recordings across those volumes however it wants. For example, you're not assigning cameras 1-5 to F:\ and cameras 6-10 to G:\. You merely assign F:\ and G:\ as avaible disk space for storage for all cameras and the system spans recordings across those volumes automatically. I think I've heard of some VMS systems where you assign cameras to specific drive letters for storage, but never seen them myself.
As for Carl's reply, correct me if I'm wrong, I don't think Carl trusts RAID groups larger than 10 disks. So in a 16 bay unit, if you break it down so no RAID group is larger than 10 disks, that means if you divided it evenlly, that would be two RAID groups of 6 disks each in RAID 6 (let's just say no global hot spare for this scenario), leaving only 4 disks of usable space in each, so that would be around 24TB out of your total of 48TB of disks in the case.
My only point is sometimes I think we need to look at beyond just what the playbook says.
I think this depends on how the VMS handles storage. I believe some VMSes write specifically sequentially to drives but most do not. Experiences here?
I assume in your scenario you are JBOD with no volume spanning software at all. If so how do you handle the balanced distribution of data amongst the drives? Assigning cameras to drives in a many to one relationship? Tedious no? And the unused, safety space you need for each drive can add up quickly. Or is there a better way that you know of?
16-bay RAIDs with one RAID group is not really advisable anyway. With 3TB drives and one global spare, that's a 13+2 RAID group or 8e13 bits. Pretty close to the 1 in 10e14 MTBF typical of hard disks. That increases the likelihood of data loss due to URE's and greatly increases the rebuild time.
A lot of times I've spec'd storage system not using RAID at all. Each disk is alone and not part of a RAID. If there's no need for the redundancy and cost is a factor, and the customer is ok with it, why not? Consider for example you have a 16 bay unit with 3TB dirves. You want super redundancy, you might do RAID6 with 1 global spare. That's 27TB of storage. If you don't use RAID, then that's 48TB of storage.
"What about safe guarding the video?" Well, if something happens to the RAID, you loose all your video. If you loose one drive in your non-RAIDed enclosure, you've only lost 3TB, or 7% of your video. If you loose 3 drives (like the 2 in the RAID 6 and global spare in the example above), you still have 27TB, or 56%of your video. And unless you have need to meet some sort of official requirement or you got big bucks , like was mentioned before, in most cases what's the chance you'll need to go back to that one portion of video that was lost?
IPVMU Certified | 04/05/14 10:38pm
Thank you for spelling out that important difference. This is the only industry I've encountered where most people seem comfortable with just using a RAID with redundancy and not a full backup. The point you've made would be very important in some workplaces.
IPVMU Certified | 04/05/14 09:26pm
Just to make it sure: We are discussing two general different things in the same topic, be careful! RAID is not Backup. Raid is redundancy mechanism, as mentioned a couple of times. A Backup is a "snapshot" of the whole disk or parts of it. A simple example should make clear why it's important to understand the difference. If you just stay with RAID, a disk failure should not affect your recordings. BUT: If a guy with enough rights just deletes Video recordings from yesterday, by mistake or not, those recordings are gone. RAID can't help you here. As you just have redundancy, the recordings will be deleted on ALL your RAID disks at the same time - bang - gone! - That's why a BACKUP could have helped...
Just my thoughts.
IPVMU Certified | 04/04/14 04:37am
Thanks very much Carl. You've provided great information and I greatly appreciate it.
For long-term storage of critical data, I've found RAID6 is quite reliable but I wouldn't trust it alone. For evidence that needs to be preserved at all costs, we rely on multiple lines of defense:
- First, we mirror two RAID6 systems in Windows.
- Second, we regularly back the data up for offsite storage.
For backup, we considered tape and I have to admit it was a tough decision but in the end, we chose to use WD MyBooks. Although tape would have allowed us to perform daily backups instead of the monthly we do with the USB drives, the cost advantages outweighed the reduced backup frequency. For added safety, we chose MYBook RAID Edition with mirrored 1TB drives but will have to search around for replacements when those fill up.
Still, we've never lost any data on mirrored RAID6 so the offsite backup is only to prevent catastrophe in case of fire, etc.
IPVMU Certified | 03/31/14 11:21pm
Hi Carl, Clearly backing up so much footage would be impractical in your case. Has RAID 6 always been sufficient to safely store the data? Are there any other strategies you would recommend to continually preserve data for a period of time? Thank you for your help.
Ok, ok. I'm not a casino so please don't put any money in my slot and pull my arm...
Anyway... To answer the question: over the years, camera counts have increased, as well as bit rates. Some may be surprised to learn that our old system was set to record at ~2 to 2.5Mbps whereas our new system records analog cameras at 2.5 to 3Mbps. h.264 is not much more efficient than MPEG2 and when we switched to a virtual matrix, we chose to keep bit rates high for both "live" and recording to get the best possible image quality.
We keep evidence clips indefinitely - some from as far back as the 1990's. We also save many clips for other departments to review so that the streams don't get overwritten. Most of those eventually get deleted. We also allow certain departments to do their own reviews but unlike some casinos, we make them clips, which they can review in a separate room. Most of them also eventually get deleted.
Many casinos provide workstations with the VMS loaded for outside reviewers and a Surveillance Agent must control playback. We find our way more efficient.
So, yes - we do retain a large amount of data, most of it relatively briefly, but it still is probably orders of magnitude less than 1%.
I am not sure your assumptions behind your calculation are correct.
Me either, For the record, I did round up carl's 7-8TB to 10TB which then became 100TB after factoring in the admittedly soft temporary footage. This would tend to skew the % number higher. On the other hand, the total recorded footage is probably higher than it should be (skewing the % number lower) because it is based on the current camera count and bit rate, and one would assume they have scaled from lower values throughout the 10 years and therefore the total recorded number is probably less than 30PB.
In any event even one order of magnitude is still 1 out of 3000 or .03%, probably higher than the overall average (if we knew what that was), but this is easily explained, as you point out, by Mr. Lindgren's strange ontological state, i.e. "Carl is a casino."
I'm thinking department store retail is probably one of the highest utilizations...*
*Of continously recorded footage, because otherwise traffic light cameras probably win easily. :)
1. I am not sure your assumptions behind your calculation are correct. It's easy to be off by an order of magnitude when using guestimate numbers.
2. Carl's a casino, which has way more significant demands for surveillance footage than the average user.
Roughly, then if 100TB had been retained a over ten year period (temporarily or not), out of (extending your bitrate numbers) ~30PB ever recorded, then congratulations, you are leading the pack in the fledgling Storage Utilization Metric (SUM), at 1 in 300 or .3%, better than the 1 in 10,000 (.01%) or 1 in 1,000,000 (.0001%) bandied about so far. :)
Not sure. I believe we have around 7-8TB accumulated over 10 years. But that's probably only 10% of the total footage we temporarily retain. Many of the clips are retained for a few months until a determination is made by the powers that be, then deleted when no longer needed.
...but we do backup our evidence clips.
Per Jim above, do you care to guesstimate the percent footage that you actually do backup over total footage?
At least in our case, backing up our streaming recordings would be nearly impossible, but we do backup our evidence clips. Why impossible? We record >1000 cameras at an average bit rate of approximately 3.0Mbps so that is nearly 3.0Gbps. Even when you divide that by the three RAID subsystems we have, that's still 1.0Gbps per system. Since we record 7/24, there is no downtime during which we could perform a backup so it would have to be continuous. That means each storage system would have to write and read 1.0Gbps simultaneously.
Obviously, the better path would be to simultaneously write to two separate systems but that would basically double the cost of already expensive storage. Our solution was to build in as much redundancy as was economically feasible and practical: RAID6 groups of no more than 9+2 disks, redundant controllers and transport and automatic failover systems. Not quite bullet proof but as close as we could get it.
"Hazard a guess if you don't mind on what percentage of all footage recorded turns out to be actually critically needed?"
My very rough guess is 1 in every 10,000 hours recorded. In a typical 16 camera system, that's roughly equal to a month (16 x 24 x 30). Even that is conservative, as rarely is there a 'critical' incident once per month in such systems.
Because the average system rarely has critical incidents and because storage loss is uncommon, most people take the risk.
Obviously, some users have more demanding requirements / needs / circumstance but this explains the general case.
IPVMU Certified | 03/31/14 05:39am
Hi Jim, I was wondering that too. However I think it's more likely that people would use a RAID level with redundancy than seperate storage for backup as the RAID will use less disks and so be cheaper. Furthermore, if you only use a seperate storage device for backup, and the backup frequency is once per day, a disk failure on the VMS might mean that the backup didn't have a chance to back up the event you wanted to see.
I didn't quite follow your last paragraph other than to think that it may be impractical to hold on to backups for years. Is that what you were saying?
IPVMU Certified | 03/31/14 05:35am
Hi Rukmini, from the point of view of backup, this industry is quite different as recordings are destined to be automatically lost after some period of time. I feel that the recordings are not deemed important unless we suddenly need them. If one has no other reason to regularly review recordings, that's probably when faulty disk media is most likely to be noticed. At the very least, I think some kind of RAID level which supports redundancy should be used.
I see you've been pie-charted by John. He's got one for every occasion :-)
Are you saying that if there ain't raid at all then they probably don't do any batch archival/backup?
Hazard a guess if you don't mind on what percentage of all footage recorded turns out to be actually critically needed?
Do you figure in 10 years, one hour is needed out of a million recorded ones? Lets say critically needed meaning in aid of LE issue or lawsuit? Do you suppose you get less bang for your buck the more cameras you have?
I wonder how many systems have 0% redundancy out there?
From Storage: Redundancy / RAID Statistics. It's 3 years old and I bet it's improved a little but the bulk of DVR / NVR appliances out there (save NAS ones) are likely to have 0 redundancy.
Video surveillance is possibly a unique use case in all the storage realm. So much writing, so little reading. Of course there are particular applications that actually use their data everyday, casinos/large banks cone to mind, and others that need or require long term archival. But for the meat and potatos of cctv from low-end sole prop to the mid-tier warehouse, >99% is just run over without ever reading and even less actually rises to the level of 'vital importance'.
Typical exchange between dealer and small to mid size customer:
Dealer: Yeah, the problem is a drive crapped out. I have a new one with me, its actually way bigger but its the same price.
Customer: Cool, thanks.
Dealer: Was there anything on the bad drive that you needed?
Customer: Wipe it. No harm, no foul.
Although this strategy seems reckless its not quite as bad as it seems since 1. The actual chance of a drive failing that actually has something critical on it is the product of the odds of the drive failing times the odds of something of vital importance happening. So you could easily go your whole life as a customer and never get burned, Murphy nonwithstanding. 2. In my exp. even drives that fail can be recovered, for a price, in 75% of cases.
So in addition to your fine questions, I wonder how many systems have 0% redundancy out there? Tho maybe nobody wants to brag about it. ;)
IPVMU Certified | 03/30/14 11:39pm
Thanks for the link to that very interesting article John. It seems ironic to me that customers deem video surveillance important enough to spend a lot of money on hardware and installation but not a little extra to secure the video that has been recorded. For the cost of one decent camera, one could buy a 4TB hard drive to mirror or back up the data from the VMS server.
I'll be interested to find out if anyone here backs up data to another device. From what you've mentioned, this seems unlikely which is a bit scary.
Related: Storage: Redundancy / RAID Statistics.
I added a poll up top about using a dedicated backup device. I think only a minority do because it's expensive and using RAID is typically enough for those who are concerned.