880TB, 2 Years, 255 Cameras - What Storage Should I Use?

Greetings esteemed colleagues,

I’m in need of your recommendations for a large and dependable storage solution for an important client of mine.

The site has 240 existing analog cameras, using Bosch encoders (H.264) to feed multiple DVRs and switcher, with a large monitoring console. The existing DVR’s, old switcher and monitoring console are being replaced with a new Genetec (client’s standard) server and remote viewing workstations. The old head-end goes away because the building in which it’s located is being demolished. Cameras run at 10fps, 640x480. Cameras and encoders are on their own dedicated fiber backbone.

There will be an additional 15 Samsung 2MP/1080p IP cameras H.264 10fps (2.56Mbps?)(client’s standard), added to the existing count for a total of 255 cameras on the system.

All cameras view an exterior, restricted-access area, record on motion, 40%, 24/7/365. Motion consists of lots of people gathering and moving around the area, 40% of the 24 hour day.

The exterior scene is an equal mix of flat grassy areas and wide concrete open areas. No trees. Cameras are attached to buildings that completely surround this area.

Client wants pricing and a design for two storage solutions: one for 12 months storage, and another for 24 months storage. Each must have 20% unused capacity for future growth. Workstations on the cctv network should have access to archived images. Having read a similar thread, maybe it’s prudent for the 2nd year of storage to be on tape?

New equipment will be installed in a dedicated room with UPS power, and dedicated redundant HVAC.

I was considering a large HDD solution by BCDVideo. They proposed 880TB storage. Any thoughts on BCDVideo?

I have to provide both solutions (1year duration and 2year duration) with budgetary pricing to the client by October 17. Any and all recommendations will be greatly appreciated!


Joseph,

The amount of storage you are estimating needing sounds in the right range.

Here's a calculation I did for 1 year:

That's about 360GB for 1 year without the padding. Add the padding and do it for 2 years that comes to the same ~900TB range.

As for BCDVideo, I started and promoted a new discussion to see what people think. We do not hear about them often and they only got 1 vote in our Top Surveillance Storage Manufacturers survey. That's not a great sign.

Given that this is a Genetec based system, I am putting out a few feelers to those familiar with larger Genetec systems to see what they use.

I've used BCDV quite a bit, theyre just white label hp boxes with 5 year warranties on the boxes and drives right from the factory... They haven't been spectacular, but I definitely don't have any complaints.

For a system of that size I would also strongly recommend Seneca Data simply from a support and design standpoint. You may have seen posts by Mike Dotson here, he is the QA/Design/Wizard at Seneca and has run more simulations and testing than I could ever imagine. Also their support dept has always been a good resource for us.

This also seems like a prime opportunity for 2 other niche options... 1) Veracity Coldstore - I have no experience here, it just seems like the perfect product for low camera count with crazy retention. 2)Contact your local Genetec SE or RSM about Genetec's new offload Archiving feature Good luck

Sean, thanks.

Btw, Joseph, a key question is how often do they expect to review video older than, say, a month? The rarer they expect it to be, the more attractive cold or offline options become.

Joseph,

I was not sure how to frame a suggestion for this so I will mention two things.

The question mentioned a long storage requirement 1 & 2 years and also that the existing head end was going away.

Seneca (who I work for and mentioned by Sean [thanks Sean]) offers two ways to help out.

  • Our XNVR line is a DAS based head end system that can have appx 100TB usable storage RAID5 in a 24 bay rackmount chassis. You would install Genetec on this system. Here is a link to our DSS Solutions
  • Our XVault line is a pure storage solution usually seen as an iSCSI target and it supports expansion chassis. Again these are 24 bay chassis. These could be your storage target for a different head end. Here is is a link for Seneca Xvault

The estimator tool we have spits out a similar value as the one John used. The tool also suggests which XNVR system will satisfy the requirements assuming a RAID5 array. A pic of the result, from our beta site, for a 1 year solution is below.

As with any estimator tool, decide if you like the resulting bit rate for a defined stream and play with the parameters or use ConstantBitRate to get what you want.

We have not yet created a tool for the pure XVault solutions yet.

With this many HDD, you would actually be better served with a RAID6 array. You would ask for two arrays made with 12 drives. We initialize these during manufacturing so that when they show up the customer does not have to build the arrays.

If you dont use R6, then having a couple of hot spares will help out.

Seneca beta Estimator Tool

Hi,

I'm ex employee of Pivot3 in Europe and I've been selling similar system, now installed and working perfectly. I can recommand Pivot3 solution, really...we installed severals casinos in France (600TB+) and the system is running perfectly. I had similars projects won also and I do not have any feedback that system is not working (even if I'm not employee anymore, I'm on the market).

If you had the possibility to contact Pivot3 people, give it a try and compare because for such solution, you will need high protection for your data.

You are right when you said that second year should be on tape...I would recommand that also as it's long time archiving.

Hope this helps.

John, Sean, Mike. Thank you for your recommendations and advice. Actually I've not asked the question about frequency of viewing archived images. The facility houses inmates so I suspect false accosations of abuse etc have to be sufficently investigated. I will ask that question Monday. Thank you!

Joseph,

I would also suggest that you look at the StorSAN storage product line. I know of several large Government instalations with large camera counts using their iScsi and Fibre storage systems. The web site is storsan.com .

In line with the requirements of such size, what is the average $/TB?

It could vary so much depending on HDD quality. 1TB Black Caviar from WD are around 90$ (MSRP) but if you take the blue line (which is not good for video) price is around 60$, so we are talking about 50% price increase. For such amount of storage, you will have a LOT of HDD, so their price is really important, apart of the storage solution of course.

As an advice, I've been playing around for long time now, I really recommand Black line from WD...do not use cheap HDD as it could cost you a disaster on the installation and more in such installation.

About price, I always said that for a professional system, robust and that could handle video streams and have good protection, you can count between 800$-600$/TB (MSRP). You could have RAID6 (or even more is you use system like Pivot3) so you will have 2xHDD protected per server, which I really recommend.

There's, of course, always cheaper solutions but I also say "cheap solution will cost you more in the future due to their lack of reliability"...it's my opinion but my experience gave me reason many times.

Hope this helps.

I am an EMC employee, so I am fairly biased in this response.

You have so many options, but EMC is the only company with a multimillon dollar lab dedicated to testing video surveillance applications in order to provide detailed design/implementation guidance. One of the many leading applications tested is Genetec SC 5.2 (also tested previous versions back to Omnicast 4.8). EMC has been at the forefront of helping the VMS applications move into the IT world for the last 8 years. This is not only for storage components, but also for running on VMWare and with RSA. The team running this lab understands internal operations of most of the VMS software and apply this knowledge to getting "Known Good" configurations for our customers/partners.

EMC Isilon was used for >25 deployments with Genetec, most as big or larger than your requirements. One of these is 2500+ cameras and has >3PB storage and the admin running it is doing this as a part time job while also being a proficient security personell.

I could bore you with technical details, but I think the whitepaper referenced will suffice and may help explain why EMC Isilon is well positioned in Video Surveillance systems. EMC Isilon Scale-Out NAS for Video Surveillance

NOTE - wrt the implementation on SAN/DAS, you have to accomodate the full 2 years per camera groups on each LUN...this can be very wasteful and tedious work in design as well as in maintenance.

Bryan,

To play devil's advocate, isn't a project of ~250 cameras and at most still less than 1PB at the low end of the Isilon scale?

In particular, it strikes me that Isilon is going to be a ton more expensive than something like Coldstore.

Can you comment specifically on estimated costs for this project's requirements?

Fair comment John. We see cost competitiveness across the board starting around 150TB and increasing from there as the system grows.

This competitiveness is mainly coming from the use of dynamic block coding on the file data versus static parity on disk data. In doing this level of data protection, Isilon can attain best of breed capacity efficiencies and use the power of each "Node" to provide the needed bandwidth/throughout for Video Surveillance workloads (which are much lower than other we work with - namely in the vide content rendering and distribution space).

As an example of this capacity efficiency, when compared to scenario with RAID6 (able to withstand 2 disk failures prior to data loss), the EMC Isilon solution can do N+3 (3 disks and/or 3 nodes failing for up to 108 disks) failing and not have any additional system overhead compared to RAID6 system for a 750TB system.

The other advantage to using Isilon here is taking advantage of the Single Volume, that Isilon software (OneFS) uses and presents to the application. By having this capability, overhead per Volume can be much smaller since one is not worried about a smaller volume needing to handle variable bit rates or unpredicted large bit rates due to low light scenarios/etc...We refer to this as the law of large numbers effect. With large retention time systems, this advantage is very helpful in saving in disks needed but also in maintaining the system.

For more data on OneFS, see the technical primer for reference OneFS Technical Overview

I really am not versed in competitor's capabilities, so this is something someone from Veracity should address. Probably in the best interest to let the system be bid by multiple technologies and determine best of breed and costs.

Repeating my specific question that you did not answer: Can you comment specifically on estimated costs for this project's requirements?

My apologies, I am not in sales and not able to provide costs, but I can provide the system details below for someone to take to their EMC account team and get pricing on:

Using Genetec, the system would be effectively (5) AND (8) NL400_144TB/24GB/2x1GE+2x10GE nodes to handle the requirements I calculated to be ~ 278 TiB/305TB AND 555 TiB/605TB using the following basic attributes and retention times of 365 Days AND 730 DAYS:

  • (240) H.264 4CIF@10fps (estimate 567 kbps per camera/stream) @ 40% recording
  • (15) H.264 1080p@10fps (estimate 3840 kbps per camera/stream) @ 40% recording

NOTE: Only 4 of the nodes need to be connected, using GE for this was tested, as was using Genetec in VMware environment. This sizing assumes use of 3 archivers.

I hope this helps.

Hi Bryan, Thank you for the information. I don't have an EMC account team to take the information you provided for a rough budgetary price. Would you do that for me?

Joseph, I work with StorSAN. We only use Enterprise drives in our storage systems. The main point is enterprize drives have special Raid Enabled firm that compensates for drive vibrations. When you have a rack full of drives the conbined total vibration can effect drive performance by increaseing read errors. The WD RE drive is what we use for all of our Video Surveillance customers. After 8+ years of shipping these drives in our systems to the Video Survellance market we have a good understanding of the expected system and drive life. The main point of failure is the drives. You should definitely use Raid 6 as suggested by others. In year 4 the drive failure rate will start to increase. I recommend you talk about a drive maintenance contract with a planned change out of all drives by Year 6 .

Forgive my silence but I'm taking copious notes! These are great responses everyone!

Thank you for posing the question. I am involved in a 480 camera deployment using 2/5 megapixel cameras deployed in a campus-type environment. We are using a "Cloud/Fog" platform running a Linux from scratch OS and a highly efficient and proprietary video streaming and distribution solution. The EMC solution sounds interesting given the scalability and ease of configuration.

Hi Joseph-

I apologize for my tardiness to the thread but I overviewed your scenario along with everyone’s responses, I thought I would chime in to offer a hand.

Like Bryan mentioned his EMC bias, I will be fairly biased in my response as well being an employee of Spectra Logic who is an innovator in the digital tape library space especially when it comes to our NVR3 solution for video surveillance retention and archive.

Since nobody has yet, hopefully I can address your question regarding the use of tape storage and its advantages.

With the NVR3 solution Spectra has a fully integrated, tiered storage solution allowing end-users with large camera implementations and/or long retention policies to manage and store their full resolution video on the most reliable and economical storage medium today……Tape.

NVR3 being a fully integrated solution consists of 3 components:

- Disk Front-end (NAS, Enterprise 4TB SAS drives, CIFS/NFS)

- VMS Software residing on the disk array allowing the NVR3 to integrate with existing VMS environments (Genetec)

o Software creates a low-resolution companion file, residing on disk (< 10% of original hi-res incoming video)

o Streaming hi-res original video to tape

- Tape Library:

o Infinitely scalable, most reliable, most economical storage (CAPEX & OPEX).

NVR3, with its low res companion file creation, eliminates the need for 90% of the disk environment. When a search of the archived video is conducted, the low-res file on disk is utilized for fast search functionality. When you have arrived on the archived low-res video clip or clips you need, start an export job to start streaming the original video from tape to a file system, FTP, burn to DVD, etc.

Lastly, in addition to scalability, cost and reliability there are far more advantages to traditional archive systems today but I will spare those for another time.

Ryan, please answer a few questions:

  • What is the cost of 880TBs for your solution? I am trying to get a sense of how it compares to hard drive based storage.
  • How do you get the video from tape? Does a person need to physically retrieve a tape and insert it?
  • How low res are the low res files? How often are the generated? Once a minute? Once an hour?
  • What if you are not sure where the video is? i.e., you can not spot in on the low res files and only know an approximate time (like overnight last April).

One of the issues you're going to run into is that all storage calculators are inherently inacurate. My internal tools are showing ~550TB for 2 year retention. Add in the 20%, and you're looking at 660TB. That 220TB delta is massive. Part of that is we really don't know how much you really need. Different environments can produce vastly different bit rates, and the difference can easily skew the price by 6 figures.

The proper way to do this is really to install the encoders you're going to use on a coax splitter and leave a test system there for a week (doesn't have to be big) to see how quickly the drives are filling. If that's not possible for whatever reason, you might want to consider using CBR. Image quality might suffer, but at least you'll know exactly what you're getting for storage.

I agree about the inaccuracy in ballpark estimating storage but I disagree with the recommendation about using CBR.

With CBR, you are either going to set it too high and waste a lot of storage or set it too low and, as you say, have quality degradation. Equally important, since bandwidth needs vary over time (and time of day), you will be wasting more bandwidth by fixing the bitrate.

I strongly second your recommendation about testing it on site, using VBR :)

Absolutely agree. CBR is useful when, for whatever reason, you absolutely can't do an on-site test but you're still required by contractual or legal obligations to provide an exact amount of retention. I'd never recommend using CBR for an onsite test - it defeats the purpose!

John - here are the answers to your questions above.

1.) Cost is certainly dependent on retention term as well as camera count but for an environment of this size given the numbers Joseph shared previously and without gathering further environmental information; I believe this would fall in the $.40/GB range. Along with that, the true cost saving is obtained in rackspace, power and scalability. With tape cartridge technology being 2.5TB for LTO6 and 10TB for TS1150, the density and the extremely cost effective scalability aspect is a great option for long retention terms.

2.) In regards to streaming the hi-res video back from tape, first off all of the digital tape cartridges reside in the digital tape library and are managed by our NVR3 software. The only reason digital tapes wouldn’t be in the library would be writing 2 copies of the original hi-res video to the tape library, then exporting one copy for DR purposes. That being said, as I mentioned our NVR3 software manages which tape cartridge the hi-res video is written to. When performing a search of the archive, the software will know which tape has that original, hi-res, video. The tape library robot will then find that tape in the library, place it in a drive and start streaming back video for you to save as you please (file system, USB, DVD, FTP, etc).

3.) First, the low-res file can be generated one of two ways. 1) dual streams at the camera level or 2) transcoded on the NVR3 system. In scenario 1 the dual stream or lower resolution is set at the camera level, so it is created instantly. In scenario 2 the low-res files can be adjusted to a high, medium or low setting. This way if you have a lower resolution (non-multistream) camera (4CIF) you can have the low-res companion file set to a higher quality and vice versa for higher resolution cameras. Lastly, in scenario 2 the low-res file creation is a customizable policy but recommended to set the policy at our quickest creation time, every 5min.

4.) The NVR3 software is quite robust in terms of searching the archive and has many options for search criteria to narrow results. Choose from date/times, individual cameras, camera pools and metadata to narrow your search. Once the search is conducted, motion detection is also displayed making it easier to locate the needed video.