Will Latency Save Analog?

Being in a industrial area of work I deal with allot of situations where real time video is really important.

You can imagine when you're stearing around 20 ton of steel you don't really want your video to have a latency in it and you end up crushing your favorite smoking stand!

I'm still quite cautious to implementing IP CCTV in these installations due to this very reason. And I'm quite wondering if this will ever be fully solved. So I'm starting to wonder..... will the latency problems with IP, save Analog from extincion ?

What are your views on this matter ?


Rogier, latency can be solved or minimize. What is the time difference you have between analog an IP?

I can see that in a few instances where Latency is a crucial factor. But I would imagine that most of the time on most scenarios, this is not an issue. Most folks will opt for the higher resolution IP has to offer even with a little bit of latency as opposed to staying with analog.

But as Hernan said, Latency can be minimized. In a matter of fact, I dont really even notice it that much unless I am really focusing on it. With a propely setup LAN with decent equipment, I dont think you will notice it much.

If your concerned about Latency, and still want HI-Def, you should look at SDI equipment. No latency and up to 1080p resolution. I havent jumped on the SDI bandwagon, but this is one selling point that it has.

While latency can be minimized with IP, with analog, latency is a not a concern at all. For the example you give of someone locally monitoring/controlling the stearing of 20 tons of steel, analog is a far safer bet. For such an application, a hybrid system might work best - the fixed cameras are IP, the PTZs are analog with direct control via matrix / cctv controller with everything recorded to a VMS/NVR.

As for SDI, you'll need SDI PTZs and potentially a new SDI or analog/SDI matrix and you'll need to verify how it all works with existing analog headend equipment. This carries its own risks and limitations.

Funny thing is, I have seen a *noticeable* latency in at least one analog system: a Dahua DVR I have on a dog-training facility. It's not a show-stopper or anything, and probably wouldn't cause a problem in Rogier's situation, but the delay IS visible if I wave my hand in front of a camera while watching on either the VGA-attached monitor or HDMI-to-DVI-connected TV.

And conversely, I've noticed with a couple different Axis cameras on my bench recently, that while viewing the MJPEG live feed in Chrome, the lag is barely noticeable - offhand, I'd say no worse than I noticed on that analog DVR, although I haven't actually measured it. Oddly, if I load up the H.264 stream via the ActiveX control in IE, the delay is a LOT worse. Viewing it through the Quicktime plugin in Chrome is downright terrible.

Again, these are by no means scientific tests, just my casual observations, but it shows latency is becoming less and less of an issue with IP.

Is a DVR really analog? :)

That video is being encoded on the DVR which will cause at least some latency in viewing on a PC monitor. By contrast, an analog PTZ directly to a matrix / monitor will have no delay.

Good point... actually, I find with most standalones and systems using hardware-compression cards, the live view does tend to be piped straight through for analog output... now that I think about it, the delay is slightly more for the HDMI output (vs. the VGA), where naturally, the system must encode the analog signal to a digital output.

The live view should be fairly latency-free but I would assume that there is a smidge bit of latency there given that it has to process multiple camera feeds and split them up all on the screen. And yeah, that makes sense about the HDMI processing as well.

As many of you may have noticed, I've been an outspoken critic of latency. That said, some systems we've tested get one-way latency down as low as 130ms to 140ms. That has changed my opinion somewhat. Our recent tests of VMS systems, encoders and IP cameras showed low enough latencies that I am willing to accept what remains on some of the better systems, assuming additional latencies aren't added somewhere else in the system like in the network or with code mergers.

Honestly though, Carl and Rogier's requirements ARE somewhat "niche". For most installations, even a half-second latency not only isn't a problem, it's practically meaningless since most people, most of the time, aren't watching live video, but looking back at recorded video, where you now have "latency" of hours, days, weeks, or even months.

Even in the case of, say, a security guard watching live... total human response time "latency" (see something amiss, decide to act, pick up the phone/radio, get up off your arse, etc.) is orders of magnitude greater than any latency introduced by the infrastructure. In the time it takes Paul Blart to get up and hop on his Segway, an extra couple hundred milliseconds isn't going to be a factor :-)

Carl, how did you meassure that time? i'm interested in that to make the same tests...

I think you have to things to check: the latency of the operation and the latency of the video itself

To see the operation latency one idea we have is to use the recorded video from the camera:

- the PTZ camera should record the operator movement (maybe press the arrow key)

- the camera moves it's the time that takes to: VMS Client+VMS Server+Network+Camera to make the movement

To check the latency of the video:

- you can record something infront of the monitor showing the recorded video.

Checking latency should be easy: put a clock onscreen that counts in milliseconds, point a camera at the screen, then put a view of the camera onscreen as well. The difference between the "live" clock and the clock displayed in the camera window will show your latency.

That's exactly how we did it. We pointed one camera at a cell phone's stopwatch that displayed time to milliseconds and another camera at both the cell phone and a monitor showing the first camera. subtracting the two times displayed from the second camera gave us latency. Granted, that is only one-way latency and there could be other factors - for instance code merger units like the Pelco CM9760-DMR or the Sennetech SCM-800-Pelco add latency in the other direction (~8ms per 2-way combine); as can some VMS' RS422/485 driver systems.

We haven't figured a way to quantify code transmit latency but it hasn't cropped up as an issue in our testing.

Carl, you are only measuring video latency. Did you check the time it takes to react a ptz camera?

Hernan,

I can think of no simple way to measure PTZ response time to PTZ control movement. I suppose it could be done using some kind of trigger and timer but the setup for that is way more complex than I have the time or inclination to do.

Besides, encode latency appears to be the primary component of total latency.

this was my idea:

"To see the operation latency one idea we have is to use the recorded video from the camera:

- the PTZ camera should record the operator movement (maybe press the arrow key)

- the camera moves it's the time that takes to: VMS Client+VMS Server+Network+Camera to make the movement"

you should measure the time in the video.

Hernan,

I doubt the results would be repeatable due to too many variables. The one-way latency is repeatable and successive frames and captures confirm that. There is a variance based on where in the scan cycle capture takes place but by using the minimum measurement over a period of several seconds, we found our measurements both accurate and repeatable.

Measuring latency from initiation of joystick movement is far less practical. For instance, at what point in the joystick's motion range do the control pulses actually start being sent? Is that a fixed point or does the potentiometer have some "play"? At what point does the PTZ recognize the pulse sequence is meant for it and respond by moving? Is that also a fixed point or does the PTZ sometimes require mutliple pulse sequences?

It boils down to repeatability of the measurement and I'm unconvinced there is any way to confirm measurement accuracy without some other trigger besides a joystick. I believe it's possible to utilize a test jig that would create something like an alarm trigger which would initiate a control sequence (perhaps a "call to preset x") but even then, there could be a wide variance in response time. For instance, we have a number of Pelco Spectra IV cameras in our parking structure and quite a few indoors. We also have numerous Spectra II and Spectra III cameras (~200 total). Spectra II, III and IV cameras all respond to joystick motion near-instantaneously through our analog matrix/monitors.

Oddly, Indoor Spectra IV cameras respond almost instantly to both joystick and "call to preset x" commands. The Spectra IVs in our parking structure still respond quickly to joystick commands but take >1 second to respond to "call to preset x" commands. What is the difference?

Carl, Some things to check to see if your network is causing any latency is to simply open a command prompt up and ping your IP cameras. It will tell you the latency in ms.

Also check to see if your cameras support a protocol called UDP which has no error checking as is a low latency protocol much better at transmitting time sensitive packets.

"UDP is suitable for purposes where error checking and correction is either not necessary or performed in the application, avoiding the overhead of such processing at the network interface level. Time-sensitive applications often use UDP because dropping packets is preferable to waiting for delayed packets, which may not be an option in a real-time system."

There is still obviously going to be some latency added in the encoding and decoding of the video but you might be able to greatly reduce the latency using UDP.

Duncan: I know a good UDP joke, but you probably wouldn't get it...

Thats a good joke.

Duncan: I know a good UDP joke, but you probably wouldn't get it...

A multicast packet walks into 100 bars at the same time.

A multicast packet walks into 100 bars at the same time.

99 bartenders ignore him and the 100th seats him in the broom closet.

Duncan, I suspect pinging will substantially underestimate the amount of latency. Pinging only measures at the network level and ignores the time to encode and decode the video, which in most LANs is likely substantially longer than the network part.

I am aware of that but a least Carl could eliminte a possible network problem causing excess latency. Carl has probably done this already anyways. On my network I am getting on average 1-2ms of latency between my Laptop and IP Cameras with 2 hops.

Carl,

Did you ever do any test determining if h.264 caused more latency then MJPEG? If so what were the results?

Duncan,

I am not attempting to "fix" anything. Our latency testing is part of the evaluation process for VMS and NVR systems to replace our existing system (Honeywell Enterprise). Latency is one factor we are testing for because as part of the system replacement process, we intend to eliminate our Pelco matrix and analog monitor wall. Accurate PTZ control is crucial to our operations. Also, although we are replacing our entire Server Room and Control Room hardware, after the process is complete we will still have ~90% analog cameras, including ~200 PTZs.

Latency testing eliminated one system and contributed to the elimination of a second: Avigilon's encoders buffer six frames of video and have other issues, resulting in >500ms latency and Pelco Endura's encoder system gave us >300ms of latency. Both systems were eliminated from the competition. In Avigilon's case, elimination was almost completely based on latency tests and our inability to follow vehicles or even people running due to that issue.

We did test Avigilon's h.264 and MJPEG streams and surprisingly, MJPEG had ~60ms greater latency.

The "best" systems we've tested yielded latency on the order of 130ms to 150ms.

I am curious to know which VMS and NVR systems are getting the 130ms to 150ms times? Do you mind sharing that information?

Lol @ Matt. THAT's funny!

I recon for my line of work, the best latency would be around 300ms or less.

To give an example, we had new Walkie-Talkies here who were digital. But we quickly found out they had a latency for about 500ms. Now imagine 2 people communicating, one stearing a 50tons crane, the other standing at the spot. He yells 'Stop' and the next thing they can do is replace a few fences which got crushed.

Ofcourse this isn't the only thing. They 'stear' the sheets of metal in the production lines and the more latency you have, the more slower their movements become if they want to precicely aim it at something. And that leads to more frustration.

I'll guess I have to try a few tests myself as Carl has done as well.
So far, for the installation I have running at the moment, I just set MJPEG and UDP so far to get the best latency.

Latency will definitely not save analog. That's like asking if color reproduction will save film photography/videography. H.264 is 10 years old and has only been mainstream in security cameras for 5 of those years, analog CCTV is 70 years old (or so wikipedia says). It is literally just a matter of time before any latency issues are worked out, dsp chips get better, software/hardware decoding gets better, etc...

There are some obvious situations right now where analog would be the only acceptable option, but lets look back on this thread 5 years from now and see what that situation is like.

IMO, the only thing saving analog is the price. But from what I have seen, as IP cameras become less expensive, the IP camera market has become bigger and bigger.

Could be a bold prediciton but I dont see Analog lasting another 10 years. Not to say that IP will be the only option, but either SDI will continue to be sold as a latency free option or some other standard will come on the scene.

Even some of the smaller Chinese Manufacturers are getting in on the IP action. If Onvif gets the kinks worked out and interoperability becomes easier and easier, I think IP will totally dominate. Also manufacturers continue to make IP systems easier and easier to implement which is also helping.

Sean,

In many cases, that's not true. Our cameras are laid out to take full advantage of their capabilities without overdoing it. If we retain the same camera locations, all we would need would be 4CIF resolution from IP cameras. Discounting the fact that 4CIF IP cameras are fast disappearing, there is no point to replacing analog with IP without leaving a bunch of holes in the ceiling to take advantage of the lower number of megapixel cameras required to cover the same areas.

Then there is the cost to fully upgrade our transport infrastructure to accommodate IP or for EoC or EoUTP devices. After our system upgrade later this year, we will still have >90% analog cameras and I see no reason to replace them with IP except for limited applications where we can take full advantage of megapixel. I foresee us buying analog cameras for many years; both for replacements as old cameras die and for new installs where the cabling is already in place and 4CIF through encoders will provide satisfactory images.

Good one, Brian :)

IndigoVision with their h.264 encoders, Dallmeier analog blades (though only on the VMS' primary screen). Possibly the Axis Q7406 and the Bosch X1600 (we'll be testing their latency with Geutebruck GeViScope next week, but they both show acceptably low latency).

Contrarily, the highest latency we measured was Avigilon through its own encoders. h.264 measured ~500ms while MJPEG measured ~560ms (yeah, I know MJPEG should have lower latency than h.264). Second highest was Pelco Endura, with >300ms.

According to Avigilon, their encoders buffer six frames of video (adding ~200ms alone).

We haven't tested IP cameras' latency, nor any VMS' latency with IP cameras because we don't care much about fixed camera latency and we intend to stay with analog PTZs for the foreseeable future though Dallmeier did demonstrate remarkably low latency in a demo of their IP PTZ (we didn't measure it, though).

carl,

why is good 130ms and bad 300ms?

Hernan,

It's all about PTZ control. Both the Pelco Endura and the Avigilon system had so much latency, it was difficult to follow moving vehicles or people running. Especially so with the Avigilon system that had the additional problem of PTZ overrun, where releasing the joystick didn't stop the PTZ. This made its effective latency >1 second.

ok, so you are just analyzing the video latency, right?

Is there network latency also?

one more thing, what is the limit of latency that you think is acceptable of operate ptzs?

I will not consider any system with >200ms latency and would prefer it be much closer to 100ms or less.

Yes, we only measured one-way latency: camera-to-monitor. That would include the one-way network latency but that should be negligible in a properly designed network (1-2ms?).

please correct me if I am wrong, but in casinos, almost all the cameras aren't fixed?

In our casino, about 20% are PTZs.

Is important to use joystick or is better to use mouse?

I'll tell you another reason that's saving analogue - Heat. I had many customers from mining companies who are not moving to IP cams due to the consistant high temps all year round. They use analogue cams connected to high temp encoders to a hardened edge switch then off to the VMS etc.

These locations vary from ambeint 35deg C to 50deg C. Course all cameras are in enclosures so your guess is as good as mine what the temp is in those but the customers trialed IP cams and weren't happy with heat related issues. Course you can't tell these guys cause now they believe all IP cameras have heat related issues.

Just thought I'd mention another reason why some customers haven't transitioned out of Analogue Cams...

I'll tell you another reason that's saving analogue - Heat. I had many customers from mining companies who are not moving to IP cams due to the consistant high temps all year round. They use analogue cams connected to high temp encoders to a hardened edge switch then off to the VMS etc.

These locations vary from ambeint 35deg C to 50deg C. Course all cameras are in enclosures so your guess is as good as mine what the temp is in those but the customers trialed IP cams and weren't happy with heat related issues. Course you can't tell these guys cause now they believe all IP cameras have heat related issues.

Just thought I'd mention another reason why some customers haven't transitioned out of Analogue Cams...

Syd, thanks for the feedback. Three years ago we had a discussion about the problems of hot IP cameras. I think that's a declining problem but some of them got scalding hot. Curious to hear anyone else's experiences here?

Hernan,

I've tried mouse and on-screen joystick controls. I still prefer a real joystick. It allows very fine control and, coupled with good variable-speed PTZs, a good operator can follow a vehicle while zooming in on a license plate or the driver. No other method of PTZ control can equal that precision. Mouse control comes second and perhaps, can be pretty precise with practice. On-screen joysticks are the worst.

In general, there's no substitute for the feel and precision of a real joystick... that said, software/mouse control does offer some possibilities that I've heard some VMSes offer (although none that I've seen), such as the ability to draw a box around a specific area of interest and have the camera automatically pan and zoom to that area (imagine a software select/crop/enlarge operation), or even to simply double-click a point in the scene and have the camera zoom on that point. In some instances, such features could be very handy.

Of course, many VMSes offer the best of both worlds, with both mouse control *and* hard joystick interface. I have an old USB flight stick connected to my bench Vigil DVR, and it works quite well, with some of the numerous controls mapped for presets, zoom, etc.

Carl, one more question. How does this systems manage the stream? Because it is not the same if you take the video from the camera or from the server. For instance, you mention Indigovision, I understand that they manage the video via multicast, so the video does not go first to the server to be regenerated.

Hernan,

Pretty much all systems we have tested route cameras (and encoders) through a server in one way or another before the stream gets to the client. I haven't dug deep enough to determine if the server actually processes the video before sending it to the client or just routes it. It doesn't seem to matter based on our tests. As far as I can tell, with the possible exception of Pelco Endura (Avigilon is another matter due to their encoder "buffering" six frames), latencies are pretty much the same on all of the best systems +/- one frame (33ms). That appears to be the limit of our latency testing resolution. We did discover that multiple samples of a latency test yielded results within that range (33ms), so we concluded that testing was only accurate to that level. We chose to use the lowest latency from multiple samples of each encoder.

Since we're testing NVR/VMS systems, we aren't aiming to get the lowest latency possible, just the lowest latency a system can provide, with the caveat that there are many other factors we are testing for. High latency is enough to disqualify any given system but that also depends on our options. For instance, we tested Axis Q7406/Q7900, Bosch VIP X1600 XF, Sony SNT-EX154/SNT-EP154, TKH Siqura S-68 E and Verint S1808e encoders with the Genetec system to get an idea which encoders give the best picture quality, lowest latency and lowest bitrate for a given picture quality. We eliminated the Sony, TKH and Verint encoders for one reason or another. We are now testing a Geutebruck system with both the Axis and Bosch encoders. Although the Axis encoder has slightly lower latency than the Bosch in comparitive testing, we have not measured latency yet - that is scheduled for the next day or two.

Our observations have told us that each encoder has pluses and minuses. The Axis encoder has very slightly lower latency and slightly better resolution (due to its D1 capability vs. Bosch's 4CIF), while the Bosch has a slightly lower amount of noise. The Axis also tends to require lower bitrates than the Bosch - Axis' bitrate "settles down" under low/no motion conditions while Bosch, even in VBR, tends to run at, or even slightly above, any bitrate caps, even with no motion. The Axis also gets the nod during high motion (PTZ fast pan, for instance). Bosch's picture gets more macroblocking and other motion artifacting and takes longer for the picture to "settle down" after motion ceases.

Determining which encoder to specify will be difficult - we'll have to weigh many factors, including cost, rack space, redundancy, warranty and support and other capabilities to make the decision. That said, comparative testing on the IndigoVision system (IndigoVision's MPEG4 and h.264 encoders vs. the Axis and Bosch encoders), led us to conclude there is no point to specifying third party encoders if we choose IndigoVision. Their h.264 encoders provided comparable, or even superior, results to the others and were priced in the same ballpark when you factor license fees, etc. into the equation. Their MPEG4 encoders fell somewhat short in many regards.

Though this is a point whoich is not part of the main discussion, I would like to clarify Matt Ion's observation "I have seen a *noticeable* latency in at least one analog system: a Dahua DVR".

The explanation is that not all DVRs are Pentaplex. The real cause of this delay is the fact that even the spot monitor on the DVR does not output the incoming signal from the camera but only plays back the most recent recording from the camera. This process of fetching the recorded image and sending it to the spot monitor can take as much as several seconds for some DVRs. In such cases, if the delay is a material factor in your operations you can either use a BNC T-connector at the the camera input to the DVR, and connect that to a spot monitor for that channel, or you can replace the DVR, if you have multiple cameras where the delay is a critical factor.