Testing IP Camera Latency

IPVM

We can test other combinations / scenarios. Let us know what ideas you have.

Reactions:

Tim Sisk

IPVMU Certified

Nicely Done...Interesting piece

Reactions:

(1)

Joe Mirolli

IPVMU Certified

Looking forward to a PTZ test version!

Reactions:

(3)

Luis Carmona

Geutebruck USA • IPVMU Certified

In a large building or campus, a camera stream might go through 2 or even 3 switches before reaching the recordering server. I wonder how much it might increase when you put another switch in between.

Reactions:

John Honovich

IPVM

In reply to Luis Carmona

I doubt that's anything significant compared to the hundreds of milliseconds essentially inherent in this application.

For instance, pinging from Hawaii to the East Coast is ~150ms and that's dozens of hops and thousands of miles.

So even if a few switches in a building added tens of milliseconds, it's probably not a factor.

Reactions:

(4)

George Whittaker

when the cpu spikes, couldnt it affect the stopwatch too??

Reactions:

John Honovich

In reply to George Whittaker

IPVM

If the stopwatch was impacted, we / you would see it on the live side but the stopwatch did not lock up / slow down, etc.

Reactions:

Pat Villerot

What an interesting way to test this visually. 400ms isn't awful, but visually noticeable. I'm pretty sure anyone can live with that.

John, since you solicited for testing scenarios:

Testing over a 802.11 bridge connection might be interesting since RF is a shared medium and collisions are simply part of the game. Also, testing over a small mesh to observe the latency build up between each node might be worthwhile testing. Both of the above would be even more illustrative if you're testing in a dense urban area where collisions are ever more common. PTZ control can be infuriating as the latency builds up. A mesh is the best opportunity to see this in action.

Reactions:

George Whittaker

could changing the stopwatch display to show the system time, and enabling timestamp overlay (from the vms) on the recorded frame provide additional information?

Reactions:

John Honovich

In reply to George Whittaker

IPVM

I am not sure what that would tell us additionally.

However, I did find an online stopwatch goes to 3 decimal places which I think would be useful. After ASIS, Derek can add some additional test runs to see what that reveals.

Reactions:

George Whittaker

timestamp probably bad idea anyway, might slow it down. thinking about it, does that mean whenever your vms has to burn a timestamp that it must decode and re-encode every frame even if it is not on live view? that might take a lot of cpu.

Reactions:

Carl Lindgren

John,

Why did you test using 1/10 second granularity? That's an awfully wide leeway (3 frames). When I tested encoders for our VMS evaluations, I used a stopwatch that displayed time in 1/100's of a second. I believe that is a much more accurate measurement.

The point is that a reading of, say 7.0 seconds "Live" could actually be anywhere between 7.00 to 7.099 seconds and a reading of 7.3 seconds could be anywhere between 7.30 seconds and 7.399 seconds. 7.3-7.099 = 0.201 and 7.399-7.0 = 0.399 so the actual latency could vary over 98% and still measure the same.

Reactions:

John Honovich

IPVM

Carl, as I mentioned in the thread above, going forward we will use an online stopwatch with 3 decimal precision.

That said, your theoretical observation is not in line with the many test runs we did. For example, for Exacq, it was .3 every single time over dozens of tests. If the actual latency varied as much as you opined, we would have seen runs where the stopwatch reported .2 or .4 but we did not.

Reactions:

Carl Lindgren

Wht would that be? Your test could have yielded: 300ms, 330ms, 380ms, 398ms etc. And, as you stated, every one would have displayed .3 seconds.

By the way, with a 3-digit stopwatch, I would bet that the third digit (milliseconds) will be just a blur. Unless, of course, you use a fast shutter.

John, of course our testing was aimed at PTZ control. We found that >200ms made PTZs tough to control when trying to follow fast-moving objects like people running and moving vehicles. Even a 20-30ms difference was noticeable.

The best systems yielded latencies between 140ms and 170ms while the worst were much higher. Pelco Endura encoders, for instance, yielded ~330ms while Avigilon encoders yielded >500ms. Dallmeier and IndigoVision were both better than 150ms.

We also tested standalone encoders and the lowest latencies were yielded by Bosch and Axis. On a related note, we also tested codecs, since the Bosch X1600XF encoder could run baseline, main and high. Adding 'B' frames and running higher-level codecs increased latency appreciably.

Reactions:

John Honovich

IPVM

Originally above, you claimed a variance of nearly 200ms, i.e.g, "7.3-7.099 = 0.201 and 7.399-7.0 = 0.399"

Now you are claiming a variance of 1/2 that - 100ms (300ms, 330ms, 380ms, 398ms etc.)

There is some variance but, even as you now acknowledge it is less than 100ms.

Again, if the variance of latency was significant, we would have had some runs where the stopwatch returned .4 or .2 but it did not.

Reactions:

Carl Lindgren

Agreed: late night - math error. I should have said 7.301 to 7.399. Still, 3 frames at 30fps.

Reactions:

Carl Lindgren

However, 300ms is not good latency when it comes to controlling PTZs. My comment to Avigilon was that with their >500ms latency (and their system control "runon", whereby upon release of the joystick, the PTZ continued moving for at least another second (500ms + 500ms), we would have trouble following a little old lady using a walker.

Reactions:

John Honovich

IPVM

I am not saying 300ms is good latency. I am saying that was what it was in our tests. As the VMS video above shows, it could certainly be even worse than 300ms.

Btw, as I think we both would agree, PTZ control is a lot more complicated than fixed camera latency because it depends how VMSes process and respond to the PTZ commands being sent, which can add even more latency to the operation.

Reactions:

Carl Lindgren

We never concerned ourselves with fixed camera latency. After all, nothing is so time-critical that even a second would matter. In fact, IndigoVision playback can be up to 4 seconds behind "Live". I believe that is due to their 4-second GOP size. We've been using the system for over a year and never had an issue with the delay preventing us from responding quickly to an event.

Reactions:

(1)

Carl Lindgren

How would you measure control latency? That was an issue I struggled with and gave up. In any event, it didn't appear to be an issue during our tests. Other than Avigilon's runon, we never observed an issue that wasn't directly related to video latency.

That said, we only tested analog fixed camera latency through encoders - both manufacturers' own and third party. Since we use the same encoders for both fixed and PTZ analog cameras, I believe our testing was relevant.

With IndigoVision deployed, we have taken the opportunity to test IP PTZs, but only by feel. IV's 9000-series 4SIF, 11000-series 720p and 12000-series 1080p PTZs all have acceptable, though unquantified, latency. My guess, based on our experiences during VMS/encoder tests, is that all three exhibit well under 200ms bi-directional latency.

We've also tested Bosch, Pelco, Sony, Vitek and JVC PTZs. The Bosch, Sony and Pelco PTZs exhibited control issues, whereby motion was not smooth and/or the PTZs also exhibited runon after release of the joystick. The best control, and overall best operation, is/was exhibited by the JVC and IndigoVision's own PTZs. Obviously, IV works with IV, but we were surprised at the poor showing of the other three.

Reactions:

John Honovich

IPVM

How would you measure control latency?

I am not sure how you can easily segment control latency from video encoding latency. I guess I would measure latency of a stationary PTZ first to get the baseline of video encoding latency and then try to measure what the latency was when panning the PTZ, subtracting the two. I am not sure if that would work though as I have not tried it (though we will in a future test round).

Knowing how the control commands are handled is tough because it is not easy to inspect. It might be that PTZ X is slow or inconsistent in sending out the commands. It could be that VMS Y is unoptimized / poor in receiving / processing requests from PTZ X, but good for its own PTZ Y. Worse, it could be a combination of both.

Reactions:

Joe Mirolli

IPVMU Certified

Let's cut to the most important piece here, Can this delay get you out of a red light traffic ticket?

Reactions:

Carl Lindgren

In reply to Joe Mirolli

Joe,

Interesting question. I wonder if anyone has ever tried to challenge a "red light scamera" ticket on that basis?

Reactions:

John Honovich

IPVM

In reply to Joe Mirolli

As long as the red light and the car are both captured simultaneously / synchronously on the same camera, it does not matter if the delay to record was 5 seconds. The video will still fairly show where the car was when the light turned red.

Reactions:

Carl Lindgren

John,

It was my understanding that red light cameras (scameras) are typically set for a delay between the red light and photo capture. The local jurisdiction was allowed to choose the length of that delay and, as I recall, some were setting it so tight that drivers who were past the trigger when the light turned red were given tickets, even though the light was yellow when they actually entered the intersection.

I seem to recall that there was a big stink about that and a number of tickets were thrown out of court until the jurisdiction lengthened the delay between light changes and photo capture. Of course, there was another big stink raised when it was discovered that jurisdictions were not following accepted standards for length of yellow light versus speed limit. In some cases, it was even proven that jurisdictions deliberately shortened yellow lights in order to maximize income.

Reactions:

John Honovich

IPVM

"some were setting it so tight that drivers who were past the trigger when the light turned red were given tickets, even though the light was yellow when they actually entered the intersection."

I certainly believe that. I am just emphasizing that it is not a video latency issue but a (bad/manipulative) policy decision.

Reactions:

Joe Mirolli

IPVMU Certified

Agreed on the red light stop camera, how about with a rolling stop, in DE they get you for failure to come to a complete stop before turning right on red. Latency could play a part with that, especially since most are connected wirelessly... I would assume the camera companies that implenet would be smart enough to realize they need full fps. I would love to find an out for these, they give you the opportunity to challenge the ticket but never surrender wasting everyone's time and money.

Reactions:

(1)

John Honovich