Specifying An NVR - Do I Need More RAM And A Purpose Built Video Server?

"One of the points of contention with the vendor is that they claim the amount of RAM on the box will not impact performance."

In general, our testing and experience concurs with that - RAM is not a primary factor of VMS performance (see: VMS Server Load Fundamentals Tested).

"however, we have crashed several of our recorders due to what appears to be memory issues."

Can you elaborate on this? You checked statistics and RAM was maxed out or? If so, was CPU also pegged as well or? And how much RAM were you using?

"I'm concerned that the Dell and/or our HP equivalent are not truly video-grade servers -- just standard application servers."

A lot of the surveillance storage vendors hype this and claim to have 'secret sauce' but most of us are skeptical of this and, yes, most surveillance vendors are just re-selling COTS machines from Dell, HP, Super Micro, etc.

Our earlier generation NVR's did fine with that number of cameras using SATA drives, Pentium Core Duo processors and 2GB of RAM on Windows XP. When we're just talking recording, you might be otherthinking it. You're not doing video "rendering", and unless you're doing transcoding (like for example converting MJPEG streams from the cameras into h264 on the server side) or higher end video analytics, it's not that resource intensive.

You should probably consult with your VMS provider what they recommend. Some VMS vendors will base performance on total pixels processed per second. So for instance, a VMS company may say they process 1000 megapixels per second recording, provided you meet a minimum CPU. So you take the total pixels of all the cameras multiplied by the framerate per second and see if they fall into that range.

Ie. 30 cameras X 2 megapixels per camera at 10 FPS = 600 megapixels per second.

You also have to get from them their estimated record rate in either megabytes or megabits per second. In the above example, that's around 115mb (megabits), or 14.5MB (megabytes) per second. So that's well under what a gigabit network, internal SATA storage or iSCSI storage system can handle, but is the VMS software capable of processing that rate in recordings?

If you consult with your VMS provider, not only do you get a better chance of properly estimating your requirements (and you can always go over a little for a margin of safety), you put the onus on them if something doesn't work right. If you "guesstimate" on your own, you leave yourself open to being accused of not following vendor specifications.

We have been developing this answer in my lab for several years now. I will mention a few things to be aware of without divulging our 'secret sauce' ;-)

The answer to this starts with the VMS itself. Different ones handle the data their own way....and will give you a different performance result. Those that have a two stage storage mechanism where the 1st stage is a high RPM drive (which acts as a cache) will be able to achive a overall higher throughput. Of course you can use high RPM drives for everything and get great storage performance, but that would not be cost effective.

The second influence is the combo of 'how many' cameras, their CODEC and the bitrate. This creates the overall payload the system must handle. Depending on where things like Motion Detection are performed (camera or server)...will affect the memory. As you get higher into the throughput numbers...the bottleneck changes from a CPU/Memory centric to a NIC/Storage centric one. The testing report mentioned by John is the low end of the throughput chain.

The usage of the NVR with respect to Clients must also be factored in. A system that is 'recording only' will have a much better performance than one where the Client function is running.

Let Task manager and Perfmon (assuming Windows) be your friends as you try to answer this for yourself.

Sorry for the delayed reply, John. Thanks so much guys! This is REALLY helpful! John: We are using 4 GB of RAM on all of our servers, and yes, it was maxing out the RAM but the CPU was only at 30%. Our vendor continues to be adamant and decidedly obstinate that no more than 4 GB should be required.

One other major issue we are experiencing is artificating and frozen frames on our IP cameras when viewing any video through the VMS client -- this is what sparked my interest in the SAS drives. When we view the stream directly from any IP camera (for example, an Axis Q6034) via a browser, the image is perfect (we're viewing it MJPEG). However, when we view the same stream and/or playback of the same stream via the VMS client, the artifacting is terrible (VMS is recording H.264). My understanding from the vendor (and this may be true for many VMS) is that camera streams hit the server HDD, and it is actually from the HDD that the client is pulling the "live" video vs a direct connection to the camera.

I apologize in advance if my above description is confusing or difficult to follow -- no sleep last night. :)

John,

I wish I could share the VMS with you, but I was told that we shouldn't based on some NDAs in place -- sorry for the inconvenience -- I know it would help.

Regarding the utilization during that specific outage:

* One VMS-specific service was utilizing about 1.2 GB and another was at about 800 MB (UVidDrv.exe*32 / CommMgr.exe*32)

* We also have a command center monitoring tool/service that was running -- Splunkd.exe utilizing 500 MB

* Other services were running between 10 MB - 100 MB (I'll have to look into what they were).

* No memory leaks have been detected, although in earlier versions of the software, that was in fact an issue.

Q6034 (as configured in our VMS) -- I verified that the settings on the camera itself (i.e. through the browser) are reflected in the VMS configs for the corresponding camera.

* Res: 1280x720

* Max Frame Rate: 7

* H.264

* Bit Rate: 1024 (CBR)

* Frame Interval/GOV: 16

* Total Recording

* The only "analytic" system-wide that we have running is "Signal Loss" notifications -- all VMD is turned off

Let me know if you need any other parameters.

Just received some additional info from one of server support teams:

* RAM was running high, but the page file was set low -- he thinks they updated the page file size to 4x the RAM. The server team is going to get me some additional logs generated during the failure, but I may not get those until next week

Thanks!

Dave

David,

Based on what you are saying I have to assume you are doing Client Activity on your Recording Server. Client activity will stress nearly everything is a system depending on the VMS. This is why I mentioned it in my post.

Yours appears to be memory stressed. This can happen if you are changing the viewing parameters to be different than the recording paramenters...and your camera does not support dual streaming. This forces the CPU to transcode the data and use up memory to do this.

You could verify this effect by running the Client on a different system and do NOT change the stream parameters.

Another thing that you could be expriencing is a poor video subsystem for a VMS that wants a good one. Based on my lab testing, the video subsystem requirements range from 'nothing special...use the CPU' to 'let the video card do some work'.

GPUZ is your friend here.

Thanks guys. I'll try your suggestion, John.

Mike,

Just to make sure I understand, when you refer to "client", I assume you are referring to running the VMS client on the server (i.e. watching and/or playing back video). We only run our clients on dedicated workstations in our operations center; however, we do have an additional component installed on these NVRs to allow our server team to monitor the health of the machine -- are you familar with iDrac cards? It may be of no consequence, but I thought it worth mentioning.