700 Camera Server / Storage Sizing Question

[Note: Originally posted as a comment in the Genetec and Milestone 500 Camera Recorders Announced report, moved here]

Evaluating the throughput rating of NVR's has been a big question of mine for some time and I currently have a design that I have been working on that has over 700 physical cameras with over 1000 camera streams (some are multi-imager cameras). Total throughput is around 12Gbps and storage is approximately 3 to 4 PB for 30 days (24x7x30).

The manufacturer of the VMS we are evaluating claims a throughput of 400 Mbps using their NVR (customized Dell server) and 300 Mbps for 3rd party server. These numbers are recording throughput and that the same rate can be achieved for video retrieval.

The client requested that the project follows their current data center guidelines and that we add to the existing VSphere environment using Cisco UCS servers and a Dell storage area network.

For this project, we evaluated the option of using Cisco UCS B Series Blade Servers (Dual CPU Xeon E5-2660 B420 M4 Blades) with one NVR instance per CPU. The Cisco UCS chassis supports VIC (virtual interface computing) network interface cards that allow for high speed fabric connection directly from the virtual machine to the core network using multiple 10Gb interfaces. The servers would be connected to a Cisco Fabric Interconnect switch that is connected to a Nexus 7004 core via multiple 10Gbe interfaces.

For the storage solution we are looking at Dell Compellent FS8600 storage controllers connected to the Cisco 7004 via multiple 10Gbe interfaces. This solution drops off the fabric at the Cisco 7004 utilizing 10Gb iSCSI to the Dell SAN. For the storage backend we are using Dell S5000 switches and SCv2080 storage arrays.

I would think the server configuration above would provide the 400 Mbps throughput rating equal to the manufactures provided server or even higher. Without having a way of testing or confirming the actual bottlenecks in the NVR platform it's hard to make any assumptions.

The other issue is the cost of the virtualized server and storage area network in this solution is a more expensive option than using the manufacturers NVRs and the management without a qualified IT department would be a nightmare. The manufacturer offers 90TB single NVRs which would provide all storage in the NVR without need of any external or attached storage. From a keep it simple standpoint the standalone NVR would be the best option.

12000 Mbps total throughput / 300 Mbps (400 minus some breathing room) = 40 physical NVRs for this project with 3600 TB of storage. Using 300 Mbps (they recommended 200 Mbps for virtualized) we would need ~60 virtual machines.

Other issues that come up would be server side video analysis and transcoding. What is the impact of storage only on the CPU and at what point of utilizing the CPU beyond this causes any impact to the storage throughput. The manufacturer being evaluated does have a tech paper showing different camera/server configurations and results that help make some assumptions but it is based on their NVR solution.

It would be nice if IPVM could start a section on NVR throughput testing and bottleneck evaluation. I would love to hear from other users here on their experience with NVR throughput, especially in a virtualized environment.

EDIT: One point I forgot to make and was a reason for my comments above. We are using high resolution high bandwidth cameras for all designs now. As such we have a need for high throughput and not high camera count NVRs. The solution I indicated above would only be 15 to 25 cameras per NVR.

NOTICE: This comment was moved from a report: Genetec and Milestone 500 Camera Recorders Announced


Andrew,

Thanks for sharing! Good topic.

Question, first, you mention 12,000 Mb/s total throughput and 1,000 total streams. That implies an average stream rate of 12Mb/s. Am I understanding this correctly?

I know you are using some multi-imagers, but even with that, a 12Mb/s per stream average feels quite high. I also understand that you are using a high resolution high bandwidth, but 12Mb/s seems uncompetitive.

Especially with smart codecs (and frankly even without), you should be far less than 12Mb/s average per stream even for high resolution.

I am emphasizing this first because if there is any way to optimize the per stream bit rate that would make everything easier / less cost on the server / storage side.

John,

Great questions. For this project we are currently specifying over 150 of the Axis Q3709 multiimager cameras (though it looks like they have temporarily stopped production). The remainder of the cameras are mainly 5mp and 3mp with some 1080P PTZs and a few Flir FC thermals.

I had originally utilized 10 Mb/s per stream but after performing some real world calculations with the cameras specified I came up with 12 Mb/s as the worst case / max.

We are evaluating smart codecs and other bandwidth savings but for the first evaluation we wanted to go with worst case to make sure we are not putting ourselves in a bind. Also the Q3709 does not support zipstream and is the largest consumer of bandwidth. I'll post the actual bitrates and calculations later today (now that a discussion has been started) when I have a chance to boot up my work laptop.

I am also not a fan of h264 compression. Even with QP of 18 to 28, the quality zoomed in to an image from H264 is awful compared to JPG. They say QP 18 is visually lossless, but I disagree. I created a presentation for the client showing the differences in compression and image quality as they are wanting high quality identifiable images.

I have multiple Hikvision, Sony, Axis, Illustra, Panasonic cameras in my lab and majority of the streams are 5-10 Mbps on 2 to 3MP cameras using high or highest quality settings. Also the Axis Q3709 doesn't make much of a difference in throughput using 0 to 30 compression, but at 40 it drops the bitrate down dramatically but also starts causing the trailing pixels, artifacts, etc from the use of compression.

I don't want to make this discussion about bitrate vs mage quality as that could be a major discussion on it's own.

Thanks for the feedback!!

Andrew, see our Q3709 test results, excerpt on bandwidth:

Depending on how dark the area gets, you may want to set a cap or else you could have bandwidth of 20Mb/s per Q3709.

Also, Axis just announced the Q3708-PVE, which is 15Mb/s max but has Zipstream and Axis says is better in low light and WDR than the 3709. So the 3709 will have more pixels but the 3708 may deliver more usable images overall (and at much lower bit rate).

Btw, what VMS are you using? That is an important point of assessing overall throughput / performance / load.

John,

I am very familiar with your testing guides. I love to read every one of them and they are a huge benefit in comparison to my own evaluations.

We will need to set caps and luckily majority of our areas have enough illumination that we will not have many dark areas.

I saw the press release for those cameras last week and look forward to testing them in the near future. I wasn't a big fan of the Arecont multi imager with movable imagers as I could never line up the appropriate views. I have not tried the new version with the integrated motorized lens. I also have an 20585PM I am testing that is 4 x 5mp. But again it doesn't come close to the Axis camera in image quality or FOV/focus range. I'm a little nervous that the Axis 4 x 5mp imager will have the same limited FOV issue.

I currently have my Q3709 set to 5mp mode as I was starting to have recording issues at 11mp. I am using Exacq for my local VMS but perform testing using direct RTSP streams via VLC. I use Elacard StreamEye for final stream evaluation using the output from VLC recorded streams.

Currently the Axis Q3079 at 5mp (2592x1944) at 10 fps indicates an average frame size of ~170000 Bytes. 170KB * 8 = 1360 Kbps per frame x 10 frames = 13,600 kbps or 13.6 Mbps with minimal motion on a complex outdoor scene.

Here are two Exacq image saves. The Arecont balance is off and I may have some compression settings off at the moment so not the best of an image from that camera. The Axis camera is set to Quality of 9 in Exacq. You can see how I can see the tops of the roofs across the lake and the top of the table on the porch. I have the Arecont tilted down to see the the table/back yard but loose the entire lake and the far/near focus has to be set at some midpoint.

Axis Q3709 Center - Day Image Outdoor

Arecont 20585 Imager 3 = Day Image Outdoor

The Arecont balance is off...

Looks like a stuck IR cut filter to me.

Normally I would agree, but this was me messing with settings. I had CWB and AWB turned off on the camera. I re-enabled AWB and here is a updated image link and a view of each imager. I indicated imager 3 on my last message but it is actually imager 2.

Axis Left

Axis Center

Axis Right

Arecont 1

Arecont 2

Arecont 3

Arecont 4

The proposed VMS that we are evaluating is American Dynamics Video Edge recorders and Victor as the client. This will be an integrated (unified) Software House CCure 9000 / American Dynamics system. Though to date, I've had nothing but headaches messing with AD and I'm not very impressed compared to Exacq, Genetec, Milestone, ONSSI. But the client is requesting the solution. Again it may just be my own knowledge of AD/Victor but I've had a hell of a time making everything work and even then it doesn't work anywhere near to the performance of Exacq.

A second issue I have is stream management. VideoEdge wants to use the 2nd stream for motion detection metadata. The VE server ends up performing transcoding of the primary stream to offer a lower resolution streams to the client. This seems to work well for the limited number of cameras I am testing, but I would like to see how well this works with a high count of high resolution cameras. Especially since transcoding is limited to a specific number of streams (They offer higher transcoding streams when using CPUs with integrated GPUs, but the Xeon E5 series do not offer this feature, and when going virtualized even if the CPU has an integrated GPU, it may still not be available).

I use the multi-streaming/multi-context in Exacq and it works unbelievably well.

The proposed VMS that we are evaluating is American Dynamics Video Edge recorders...

Why is the client so set on these recorders? Exacq has appliances as well, of course. Is AD substantially cheaper or just have better integration into SH, or better sales people?

Since Exacq and AD have common owners, maybe you can go up the ladder at Tyco...

The Software House and American Dynamics platforms have become a single unified platform which is more than integrating two systems together. This allows for a single system update of both Access Control and Video Management versions in the same process and tighter integration of information across a single database.

There has been many issues with integration and system upgrade incompatibilities. The client had an integrated system previously that had been broken over time and is no longer integrated. They do not want to get burned again dealing with updates between multiple manufacturers. You will still have issues with firmware versions between the VMS system and individual cameras.

I do not anticipate Tyco bringing Exacq into the Unified platform (this is just my thought, I have no insights on the Tyco/Exacq roadmap). CCure 9000 currently allows 3rd party integration since v2.2. AD had also worked with Exacq in the past as they OEMed the software for the hybrid recorders. So there is history prior to the Tyco acquisition of Exacq.

http://www.swhouse.com/products/Video_Integration.aspx

VideoEdge and Victor is an enterprise level application with many other features that Exacq does not provide that we would want for this client. There are many trade offs. I just find Exacq an easier system to configure and to use.

I appropriate the comment.

I can't find the edit button for my reply's. I meant to say I appreciate the comment. That gave me a good laugh to re-read.

There are a few other typos I'd like to correct, but my quick use of spell check on my last line picked the wrong word. :P

lol, i was typing "please do" as a response. ;)

In any case I doubt that the VE boc is going to do much more than thrash if it has to do transcoding and VMD for all the cameras.

As a second discussion. I'd love to hear anyone's input that has experience integrating CCure 9000 with Avigilon, Exacq, Genetec, Milestone, ONSSI as those are the currently supported systems.

Anyone using Victor as the main client? If not, what client configurations? CCure 9000 Client + VMS? Experience/issues running both clients from the same workstation? Or do any of the VMS solutions provide a different single user interface (Genetec Security Center?).

This is some of the original research I was going to perform prior to the VMS being pre-seleted by the Client. In the future this discussion may still come up, but for now I haven't been able to fully research all options. They currently have CCure 9000 and this will not be changing anytime soon.

A few additional comments/design info before I get around to posting the calculations.

I am using 10 fps on the 11mp imagers and 15fps on the other fixed cameras. For the PTZs I typically like it to be 20+ FPS due to constant motion. We may drop these frame rates down for additional savings as well as utilize motion based recording in non critical areas.

But for the overall discussion I'd like to keep it regarding server throughput vs actual camera throughput.

This job seems like a 'squirrels nest' for lack of a better term. I bet you could find completely different job to do, with a better VMS solution, spent a hell of a lot less time and stress + make more money.

Walk away from that hot mess and let those 'engineers' that are butting into your space do it them selves.

FROM: Learned it was better to walk away from some of them long ago

While the job could be a 'squirrels nest' that is why I am asking these questions here for other professionals in the industry to provide feedback.

This is a long term client and they have selected us based on our past qualifications and many years of working with them. There is no benefit to us as the engineer/designer on the project to provide a solution based on the teamed integrator or sales representative making a larger cut. The team was selected to provide a valid solution that is in everyone's best interest.

There are no other 'engineers' butting in. I have expressed general concern that there may be better VMS applications than American Dynamics/Victor but I do not have any specific reason they should not select the product over 3rd party integration, especially since they have been burned in the past as I indicated above.

My job is to support our client and to provide a design that will work for the long term. They are open to our recommendations but I need to have substantial information to provide as a reason why they should not go the route they have chosen.

Tyco and American Dynamics have been around for a long time and have been great fairly good at providing responses to issues for our Clients in the past. Sometimes certain issues seem like they should have never existed or they should resolved faster, that is no different than any other manufacturer we have worked with. Everyone within Tyco, AD, SWH, etc and their manufactures representatives have always been very helpful to resolve questions and concerns. Again no specific reasons to not select the Tyco suite of software for this client based on that.

While I appreciate your response, it was not very helpful as no technical insight was provided indicating why I would want to walk away from a long term client. I do not want these discussions to be emotional nor do I want to show speak negatively about any manufactures in general terms. Everything should be based on technical reasoning.

As a distributor, do you offer any solutions or is everything indicated in competition to what products you sell?

I do not anticipate Tyco bringing Exacq into the Unified platform (this is just my thought, I have no insights on the Tyco/Exacq roadmap).

Andrew, you probably know more than me about this but I have assumed that Exacq is part of the Unified platform. For example, from the "victor Management Software for Unified Security" page:

From a single, intuitive interface, victor lets you manage live/recorded video from all of Tyco Security Products recorders - Intellex DVRs, ADTVR, VideoEdge NVRs/Hybrids, and exacqVision. victor even takes unification further with single point event management with Software House’s C•CURE 9000 access control system. (emphasis added)

Am I missing something?

You can bring exacqVision into Victor but I was thinking it was more of an integrated vs unified method. Though since Exacq doesn't have a central database so maybe Victor Site Manager becomes the central database and as such exacqVision is considered part of the unified platform. When I think of Unified I you have the option of installing CCure and Site Manager at the same time. Though I guess you add Exacq NVRs to the Victor platform the same way as adding VideoEdge recorders. So you may be correct.

I will follow up with Tyco and see how they define it.

Andrew, I forwarded this to product management at Exacq and AD asking them to comment. People are still recovering from ISC West so it may take a few days but I think they can help clarify both the platform choice and the server load element.

I'm going to try and add my Exacq server to the Victor server I have running at the house. Strangely it shows the Exacq windows version on my laptop but not the Exacq linux virtual machine I am running as my NVR.

I was able to add Exacq into Victor. But now that I think about it I am still unified as American Dynamics Victor is the unified client. That doesn't help use the Exacq client. I need to research what information could be passed back into Exacq if it is added to Victor, but then not use Victor. Probably defeats the purpose of Victor.

it's is not much point to speccing what xyz hardware can handle, because how much you can do with x amount of hardware depends on too many factors...

1, how your cam views are built

2, how many concurrent viewers of which views

3, Analysis impact (motion detection etc)

4, how much expansion you want to have

5, how much you want to over spec the system so it's faster (you can have a system that's 50% loaded and fast, or 90% loaded and slow as a wet week)

Frankly if they have their own internal IT who are going to "babysit" this system then fairly simply let them use whatever they want, that way if it's not enough they can sort it out themselves.

it's is not much point to speccing what xyz hardware can handle, because...it depends on too many factors... simply let them use what they want.

The man who says he can and the man who says he can't... are both correct - Confucius

These are all very good questions and ones that I also brought up in discussions with the client.

Now that I'm thinking about my original post (or comment to a different article), I'm guessing my original thought wasn't what John had made the new topic. I actually didn't need assistance per say for this system design, but it would be nice in general if IPVM staff or users could provide real work experience of max throughput of different VMS systems. And if anyone possibly had any experience using the Cisco UCS platform and Virtualized NVR's and any throughput impact when using multiple 10gbe interfaces and fabric connectivity.

Then my second question which I may move to a separate discussion was the integration of CCure 9000 with other VMS's and users experiences and if anyone is using Victor here.

To answer your questions, thankfully the client is looking for a limited number of views and only 9 cameras per view. They do not plan on watching a majority of the cameras.

I'd prefer on camera video analysis vs using the NVR to perform analysis and transcoding.

Thank you for a thoughtful discussion and good questions.

Feedback from Tyco on the VideoEdge / Exacq tradeoffs:

Integrated vs Unified

When referring to Integrated versus Unified, at a high level, Integrated refers to live/search user experience, where Unified additionally provides a shared administrative user interface.

  • Unified example:

VideoEdge is developed in parallel to victor releases and is intended to offer full support for all VideoEdge features, i.e. it’s VideoEdge’s native UI. This means you can fully configure VideoEdge NVRs, provision cameras to VideoEdge, manages users, visualize VideoEdge analytics and manages alarms all by the victor client. It is also true that victor & C-Cure now share a common server so the same is true about provisioning of C-Cure 9000.

  • Integrated example:

Although Exacq NVRs are supported by the victor platform most functionality is around viewing live and recorded video. victor is not designed or intended to be the primary UI. There is no exacqVision configuration and no exacqVision user management. Alarm/event notification is one way communication to victor and handled alarms are not passed back. Exacq releases are not timed with victor & VideoEdge release cycle and feature support is driven by market feedback a customer prioritization

Write to Disk Throughput Specification

VideoEdge software only: Up to 300mb/s. VE Micro: 50mb/s, VE Desktop: 100mb/s, VE 2U NVR: 200mb/s, VE 3U NVR: 300mb/s, VE Rackmount NVR: 400mb/s

  • Theoretically throughput higher than 300 mb/s can be achieved on VideoEdge software only with 3rd party hardware. However since we do not provide factory testing of 3rd party hardware and given the mix of components and plethora of ‘COTS’ hardware we limited our performance guidance to 300 mb/s. System integrators can always test actual performance and design above 300 mb/s but are doing so at their own risk.