Subscriber Discussion

16 HD Cameras 3 Displays On Milestone, Video Display Quality Suffering, Any Suggestions?

UI
Undisclosed Integrator #1
Aug 03, 2016

I have a client with 14 ip cameras most are running at 2048x1536 10 fps, some a bit less. I'm using Milstone xProtect with a purpose built server 4th generation intel i7, 16GB RAM windows 7 and a Radion 7950 HD video card with 3GB ram (card is a couple years old) . The video images ( varying amounts ) are being sent to 3 displays. Performance is fairly good but he is complaining about some jerkyness to images and wants an improvement .

This customer now wants to add 4 to 6 more camears to the system. I think it has the processing power, but I don't know. Does anybody here have experience and systems like this? Do you think that a new top-of-the-line video card will help? i've looked at some of the commercial systems they don't have specs anywhere near what I built. I tried one of the milestone husky systems on another job, and was really just pointed disappointed it couldn't handle the video at all.

Thoughts or recommendations would be appreciated.

Thanks

Bill

JH
John Honovich
Aug 03, 2016
IPVM

Before offering any recommendations, I just wanted to clarify if you are using the server for both recording and live monitoring? I assume you are given the references to the video 'being sent to 3 displays' but want to confirm.

I make a point out of this because running the server software and doing multi-screen live viewing on the same machine can create issues (as live displays can be quite processor intensive).

Also, what version of Milestone are your using (the newest 2016 version claims live view processing improvements)? Finally, do you know if you are using multi-streaming for the video displays? That could reduce load.

(3)
UI
Undisclosed Integrator #1
Aug 03, 2016

Yes I'm using it as a recorder and a display device. It's milestone Express 2014. The video card support SLI it has HDMI out and the two additional monitors are on DisplayPort. We bought a multi-year license so I don't think I upgrading

it would be a big deal this customer wants performance more than savings.

UM
Undisclosed Manufacturer #3
Aug 04, 2016

I think John hit the nail on the head when he asked you about multi-streaming.

I am not familiar with Milestone, but the way to solve this problem with VMSs in general is to make sure the VMS has two streams to each camera. The full 2048x1536 10 fps, and a second stream at, say 512 by 384 at 10 fps (for example...). The VMS client will then automatically select the stream that will best fit the display size, which on a 5 by 5 screen should be the lower resolution stream. You might have to play around with different frame rates and resolutions for the second stream. Do this correctly and you'll see a 80 to 90% drop on the CPU burden and the jerkiness should go away. In the case of the figures above, the price you pay is a 12% increase in bandwidth from the server to the cameras.

Without multi-streaming not only is the burden on the CPU is very high due to the h.264 decoding, but also the image downsizing operation can put a lot of pressure on the GPU memory, depending on how it is done.

(1)
Avatar
Jon Dillabaugh
Aug 03, 2016
Pro Focus LLC

If this is simply a client PC, your specs should be fine for viewing many more 3MP camera streams. But, if it is also the server (as John mentioned above), that may be the root of your issues. You may consider a stand alone workstation for driving the displays.

Other areas of worry for me would be:

1) NIC connection speed. You will likely need a Gigabit connection on the server AND viewing station in order to view this many streams.

2) Total switch throughput. Does your switch (switches?) have the ability to support the total bandwidth needed? Are the server and client on the same switch as the cameras? If not, how fast is the uplink between switches? Are they managed and have you checked throughput?

3) Camera feed. Have you viewed the cameras directly (web page) and checked to see if the framerate is being delivered at the appropriate rate? If not, maybe the cameras aren't able to support the resolution and framerate asked specified. We have seen some cheap Chinese cameras that couldn't keep up with their published specs. You didn't mention brands, so this may not be relevant.

4) Codec. Are you using MJPEG, H.264 or H.265? Some codecs will be more processor intensive than others. H.264 or a smart variant will be the least intensive for your display PC. MJPEG and H.265 will be more of a processing load when compared to H.264 (any variation, smart or not). Try changing your recorded codec to H.264 and enable smart codecs if possible.

5) Recorded or live? Is this jittering only happening in recorded video, or does it also happen in live viewing? If it is just in recorded video, you may need to assess your server storage. If you are writing to a single disk, you will need to be sure it has the ability to write at the speeds necessary. If the drive isn't a surveillance based drive, and it does not have the proper cache and spindle speed, it may be to blame. 14 - 3MP cameras using H.264 should not be an issue for any of the surveillance style drives. You should be able to record at least up to 32 cameras on a single surveillance drive. Also, make sure you aren't using a single drive for both the OS and storage. You should have two drives minimum, one for the OS and one for video storage.

There could be other, less likely issues, like malware and additional remote viewers, but going over every single scenario could take a lot more time to discuss.

(2)
Avatar
Ari Erenthal
Aug 03, 2016
Chesapeake & Midlantic

While your computer is fine for recording all the existing cameras and more, I don't know if you have the resources to display to three monitors at the same time. Maybe upgrade the graphics card? Take a look at the GTX 1060 or GTX 1070.

MM
Michael Miller
Aug 03, 2016

As others have stated I would sell them a dedicated client machine with the specs to drive 3 or more monitors. Client/server setups always have some type of issue.

UI
Undisclosed Integrator #1
Aug 04, 2016

Thanks for all the comments I wasn't aware that the server and separate client machine was the way to go. I thought not having to stream all cameras from server to a client was a plus.

The Server setup I built had a really good built in dual gigabit NIC. The camera set up is between 2 buildings 300 feet apart I have D-Link 1210-28P's in each building each having 12.8 Gbps capability the two switches are connected with CAT6A cabling and all cameras are wired with CAT6.

Cameras are mostly Chinese HIKVision and Dahua with H264 I have a couple of Mobotix 720P cameras (great image for 720) and 1 Sanyo VA94s HD PTZ. Probably putting in more Chinese stuff due to price (I can have 4 of these fail for the cost of one competitors camera).

Jittering is only on live picture, playback is great, so I am leaning towards the suggestion of the GTX-1070 or 1080. This customer has money and wants performance above all, we have had several incidents at his shop and law enforcement so far has been very pleased at what he gives them for evidence. Law enforcement is also a big portion of his business.

One question, if I build him a client machine and use the GTX-1070/80 do I have to build a lot of processing power in the client also?

Avatar
Jon Dillabaugh
Aug 04, 2016
Pro Focus LLC

I don't think you will need that big of a video card for simple video, as long as you have a seperate client PC.

We drive four displays with much less. Make sure you have the best CPU you can afford, i7 quad or better, 16GB RAM, and a decent graphics card with enough outputs.

Again, make sure you have a gigabit connection to the server and workstation. Verify that it is connecting with gigabit, don't take it for granted. We had issues in the past where our workstation reverted to FastEthernet due to IT misconfiguring a switch.

And for a client that has money to spend and wants quality, I would strongly consider upgrading to better storage drives. Those Caviar Blue drives won't last long in this usage. Upgrade to Purple drives or better. It isn't even that much more expensive.

U
Undisclosed #2
Aug 04, 2016
IPVMU Certified

But, if it is also the server (as John mentioned above), that may be the root of your issues. You may consider a stand alone workstation for driving the displays.

Normally I would agree with y'all, but I think in this particular case he might actually do better by bucking the conventional seperate viewing client/recording server paradigm.

Why?

Because his problem is 'jerkiness'.

First, everyone is right to say that live viewing places a huge burden on the recorder, it absolutely does!

And in fact, (assuming no server side analytics), on a per channel basis, I would say it's at least 2x the cpu resources to display vs record. Do you agree?

So, having enough cycles for the recording process should be easy, as long as we are over-building the display capacity.

Adding another physical network hop to the video delivery is unlikely to make the display less jerky.

It's not the recording process that's starving the display, it's that the display is underpowered.

Therefore, IMHO, he will get the most smoothiest, jitter and judder-less video by using high-end video and faster drives.

To say it a different way, because he wants to see all the channels that are being recorded, a seperate machine even for viewing only, would likely be the far more demanding one.

Also, he can take it one step at a time, build out a single high performance machine with high end video and fast drives. If it's still jerky then build out a smaller recording machine and move the drives.

UI
Undisclosed Integrator #1
Aug 04, 2016

Forgot to mention that the operating system an Milestone are on a ssd drive and all recording and archiving are on (3) WD Blue 3Tb drives.

DG
Donald Gordon
Aug 04, 2016
IPVMU Certified

Bill,

I've recently completed a major installation with Hikvision digital cameras and Milestone XProtect Corporate 2016 R2. My testing and benchmarking indicate that your graphics card upgrade is likely not to give you a boost in performance. The graphics card rendering of the Smart Client is not that resource intensive. Instead, your problem will be decoding all those H.264 streams that are sent to the three monitors. The Smart Client uses the host based CPU to decode the stream, not the graphics card. Only a faster processor with more CPUs will help. BTW, the Smart Client is presently limited to supporting only up to 8 CPUs for decoding.

Since you are using an Intel I7 processor, you might be in luck with the Intel Active Synch HD Graphics that the Milestone Smart Client supports for 2016. This offers hardware acceleration which provides significant performance improvements. Check the link Milestone provides to the Intel web site to determine if your processor will support the hardware acceleration of the H.264 decoding.

We are also supporting 3 monitors in our configuration, with up to 25 cameras per monitor on each monitor. This is a very high end workstation but does not have the CPU resources to decode all the H.264 streams. Milestone will allow you to view MJPEG streams for live viewing and recording H.264 streams for playback. This combination allows us to provide the 75 cameras per workstation during live viewing. We also use the Milestone rule engine to elevate the frame rate on video motion detection. We also offloaded all the video motion detection onto the cameras instead of the recording server.

The combination of all of these techniques allow us to display 75 cameras per workstation. We would rather not use all of these techniques but until the Smart Client supports decoding on GPUs or supports more threads using host-based decoding, this is our only option.

Good luck with your upgrade.

(3)
(2)
UI
Undisclosed Integrator #1
Aug 04, 2016

Thanks for all the great advice, sounds like I will be building a server and upgrading to the newest MIlestone software to take advantage of the new video capabilities.

Does anyone have a recommendation on the server motherboard/cpu combo? Also any point in the server putting in any graphics card at all, other than on board or a cheap one?

UM
Undisclosed Manufacturer #4
Aug 04, 2016

Here is a list of the Intel CPU's that support Quick Sync. Milestone XProtect V2016, V2016 R2 are able to hardware decompress H.264 using Intel Quick Sync.

(1)
JH
John Honovich
Aug 04, 2016
IPVM

Milestone feedback: "Try to upgrade to the newest 2016 R2 Smart Client. Using the built-in GPU part of the Intel CPU’s, the Smart Client performance is increased up to 5 times." Here is a whitepaper that goes into more detail on that.

Also, as for Husky, "Husky is not designed to be running as a client. It is meant to be running as the recording engine."

Avatar
Brian Selltiz
Aug 04, 2016
IPVMU Certified

I would do everything Donald Gordon recommends. Buying a separate viewing machine that has an Intel processor which supports Quick Synch will go a long way in improving your situation.

Also, check to see if your WD blue drives are 7200 rpm which I don't think the 3TB models are. 5400 Rpm drives will degrade performance and not last as long. I would recommend the WD RE series.

Avatar
Jon Dillabaugh
Aug 04, 2016
Pro Focus LLC

The Caviar Purple drives are all 5400rpm drives and they are rated up to 32 cameras. I'm unsure why you'd say they aren't appropriate for this use?

We have used them in many systems, with more cameras than he's suggesting here, and have not had a single issue with the Purple drives to date.

Avatar
Brian Selltiz
Aug 04, 2016
IPVMU Certified

Hi Jon,

They may work fine for you with your configs but Milestone recommends and I agree that at 7200 rpm drive will perform better than 5400. Intellisync makes it even worse if purple drives use that like the green drives do.

The RE drives are enterprise class and having proven to be extremely reliable.

John recently did an article on integrator drive choices, probably better to reference that than go by my recommendation anyway :-)

MG
Michael Gombos
Aug 04, 2016

You may be having an issue using that GPU as Milestone doesn't support hardware acceleration via discrete GPUs. Make sure you are running the latest smart client and see if hardware acceleration is running. I'm betting that it is not. Also, what operating system? HW acceleration won't work unless it's Windows 8 or Windows 10.

I have a system running 21 1.3MP cameras recording and live viewing and it's fine with HW acceleration. It sits around 40% load and it's only an i5 CPU.

(1)
UI
Undisclosed Integrator #1
Aug 04, 2016

Arrggg, not Windows10; but if that's what it takes I'll do it. Is the client video viewer separate from the server software, I guess what I'm asking is do you have to have a matched set. Can I use express 2014 with client 2016? I guess it doesn't matter if I have to run Windows 10 to get the advantage of the new video capabilities.

MG
Michael Gombos
Aug 04, 2016

Yes you can use the 2016 viewing client with 2014 server. The reason it won't work with Windows 7 is due to Intel Quick Sync not being compatible. You may get into a situation where you have to disable that GPU. You could always reach out to Milestone support as well if you have difficulty getting it to work.

To verify it's working once you update the OS, click on the gear in the upper right hand corner of the viewing client, click advanced, and then under the Video diagnostics overlay dropdown, choose Level 2 and click OK. Your cameras will have an overlay on top of them and it will say Hardware acceleration: yes or no.

Avatar
Mike Dotson
Aug 04, 2016
Formerly of Seneca • IPVMU Certified

To add to Mike's comment on the 2016 Client, I have also been testing the performance in my lab using 1080P at 25FPS streams.

On a high end I7 that has HD4600 GPU and a good video card...and Win 8.1 or newer.... I am able to see 25 streams on a single monitor.

If I use more monitors...the total stream count drops off.

Avatar
Jon Dillabaugh
Aug 04, 2016
Pro Focus LLC

While we don't use Milestone, we do have clients with multiple viewing stations with 140+ camera streams across 4 displays driven by a single GTX 750 graphics card using DW Spectrum. A grand majority of the cameras are MP IP cams, but there are still about 50 analog D1 cams mixed in.

Spectrum maxes out at 64 cameras per view (window). We have four displays, each with an iteration of the client loaded. Each view can have up to 64 cameras live or in playback simultaneously. That is a peak of 256 streams in this config. The client has around 140 physical cameras at the moment, so we can't say for sure the impact of having 256 streams live. We can open the same 64 cameras four times, but I'm not sure that proves anything.

Avatar
Mike Dotson
Aug 04, 2016
Formerly of Seneca • IPVMU Certified

I can believe this. I do performance testing on a variety of VMS Servers and Clients....and each one has its own peak performance on a static set of hardware.

The Pelco Client can also run multiple instances on a system and get a higher total stream count.

UI
Undisclosed Integrator #1
Aug 04, 2016

A lot of great info, I am planning on pulling the graphics card out of my server/client and making it a dedicated server with just the on board graphics and using it in the new client.

The current cpu is an i7-4790 which does support quick-sync but not sure if that's necessary in the server or is more important in the client?

I wish I had know about the windows 10 thing last week as the upgrades were free through last Friday, oh well. I miss stated earlier about my drives, they are all WD Red's, not sure it's worth ditching for the purples, but 3TB purples are only $110, would just be loosing 30 days worth of data (guess I could clone drives onto purples?).

Anything else other than the MIlestone version upgrade?

Avatar
Jon Dillabaugh
Aug 04, 2016
Pro Focus LLC

Keep the Reds. They are just a faster version of the Purple.

JH
John Honovich
Aug 04, 2016
IPVM

The current cpu is an i7-4790 which does support quick-sync but not sure if that's necessary in the server or is more important in the client?

Client, because that is where the Smart Client will run and display/decode video.

MG
Michael Gombos
Aug 04, 2016
(1)
UI
Undisclosed Integrator #1
Aug 04, 2016

Perfect, will be building this by the end of next week.

Avatar
Mark Kirkeby
Aug 08, 2016

For now it is still possible to upgrade to Win 10 free.

I used a Media Creation tool USB drive and installed on an offline machine, it authenticated when I brought online, definitely worth a try

U
Undisclosed #2
Aug 04, 2016
IPVMU Certified

Is there server side motion detection?

If not, or if you can use keyframes only, the recording CPU load shouldn't be an issue.

UI
Undisclosed Integrator #1
Aug 05, 2016

The existing Intel i7-4790 is a quad core, since I have to buy a new cpu for the new client I was looking at the i7-6700 which is also a quad core. My question is, would there be any reason to pay 3-4 times as much for an 8 core and where would the 8 core show the most benefit, in the server or the client? What about a real server motherboard and a Xenon vs desktop and i7?

Thanks Again

Bill

UI
Undisclosed Integrator #1
Aug 05, 2016

I just noticed none of the 6 or 8 cores support quick- sync, so I guess they are out of contention on the client side.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions