Cameras Unresponsive In Extremely Complex Scenes

IPVMU Certified

I have a temporary wireless camera setup which I will deploy for a recurring event every year. It is the largest county fair in Indiana and the scene is very complex at night. There can be over 30,000 visitors, several rides, flashing lights, flags blowing in the wind, and a dark skyline, although more than 90 percent of each scene is occupied with constant motion. The cameras consist of one (1) Axis P3707 Multi-sensor and four (4) Axis Q6055 PTZs. They use Ubiquity Access Points which connect to a Dell (an older loaner model) computer and then run to monitors in a Mobile Command Center approximately 300 feet away via Cat6. There is a PoE Extender at about the 200ft mark. The integrator did not use a VMS for the maiden voyage- due to the last minute request on our part - and instead opted to use browsers for viewing with on board SD cards for recording. The Access Points are all close to line of sight without significant obstructions and showed full signal strength. Each camera had its own AP and there was a single AP at the server. We later added a second AP at the server to split the bandwidth, with no improvement in performance. The cameras display very little latency during the daytime (when motion is limited) with just a little more at night when motion is nonstop. However, controlling the PTZs at night is nearly impossible as they can take up to 20 seconds to respond - especially on the camera furthest away from the main access point (400 ft). Due to the scenes being completely occupied with motion and the scene complexity, I doubt that any Smart Codec adjustments will have any impact, but I've tried adjusting everything from frame rate, to compression, to I frame interval to no avail. Sharpness adjustments also failed to correct the issue. The system actually seemed more responsive in MJPEG, but that may have been a fluke as it seemed to be fine one minute and then perform terribly as soon as my boss showed up to view the system or we were looking for a missing child. These cameras need to be responsive when children (frequently) get lost in such a large crowd or we are tracking criminal activity from camera to camera. THe intention is to eventually rely on analytics to speed up these searches, but we are no where near that at this point. Have we exceeded the ability of wireless in this case or could it be an under-powered server that can't compute fast enough to keep up? I have time to cable the project this year, but it was set up on two-weeks notice last year so wireless was the only option. I would, however, like to continue to test the wireless concept as we are considering building mobile units for critical incidents in the future. I will be adding several 2-5mp cameras to the system via cat6 and running it on an Avigilon VMS once we work the bugs out this year. Any suggestions or advice is much appreciated.

-Dennis

Reactions:

Undisclosed #1

IPVMU Certified

Is the picture latency delayed, or just the control to the PTZ?

Put another way, if you were close enough to the camera to see with your own eyes when the mechanism actually moved, would it still be 20 seconds, or is it that you just see the results of your action on the screen after 20 seconds?

Have you tried disabling local recording?

Have you tried using a slower frame rate/lower res setting?

Are the access points directional?

Have you done a wireless survey?

Reactions:

(1)

Dennis Eaton

IPVMU Certified

I believe it's a delay in PTZ control because of the drastic changes in the scene by the time it moves. The scene isn't just delayed. People are completely gone from the scene by the time the camera responds. I haven't tried disabling local recording. Can you explain how that helps? I've tried lower frame rates and lower resolution to no avail. Access Points are directional. I'm not familiar with wireless surveys, but I will be researching it now.

Reactions:

Undisclosed #1

IPVMU Certified

I haven't tried disabling local recording. Can you explain how that helps?

Just that the CPU burden of local recording (however much that is) would increase with scene activity, because the size of the p-frames needing to be stored would increase, as it encodes greater differences between frames.

That said, I doubt its significant enough to make much difference.

Axis cameras have a server report and system overview that will tell you information in real time about cpu usage, concurrent network connections etc. You should run this to see how it’s performing during the critical periods.

Though my guess would be the network. Can you tell us more about the radios, which models, distances and topology?

Reactions:

(2)

Dennis Eaton

IPVMU Certified

I will have to dig the access points out of storage tomorrow and I will Post a model number tomorrow evening. They were set up in a star topology. Initially we had set the system up with five stations reporting to a single access point. We eventually split the system into two access points, but that had no effect on the performance of the system. Also we are recording to SD cards at the cameras, hopefully eliminating any impact on the CPU. The distances between stations and the access points range from 25 to 125 yards. I will probably have the system reinstalled as soon as the weather breaks here in the Midwest, so we can Start trying some of your suggestions. I will probably just leave the cameras up permanently once we get it figured out.

Reactions:

Undisclosed Distributor #2

What about removing the PoE extender at 200ft and changing it to a normal switch?

And the bandwidth? Is it VBR or CBR?

How much bandwidth is comming in? At day? At night?

CPU use at the client?

more responsiveness in MJPEG is normal because there is less encoding and decoding to be done.

Reactions:

(1)

Dennis Eaton

In reply to Undisclosed Distributor #2

IPVMU Certified

You might have a point with the POE extender. I have a viewing station at the CPU in our substation that seems to have less problems controlling the camera and then the 300 foot line with the POE extender running to a mobile command center. The mobile command center is experiencing the lion’s share of the problems. I ran the system on VBR, but I don’t recall What the bandwidth readings were. I would have to imagine they were through the roof considering the scene.

Reactions:

Tony Darland

In reply to Undisclosed Distributor #2

IPVMU Certified

UD2 - All things being equal, will performance be better with a switch vs an extender? I'm curious.

Reactions:

Michael Miller

If setup correctly you shouldn't be exceeding the bandwidth of your UBNT wireless with the 8 camera streams your currently using.

Do you have anyone that can confirm that the wireless is setup correctly?

Do you have this issue when there are people in the complex and when it's empty?

If your using 2.4 or 5g for your wireless (which you most likely are) you could be dealing with a ton of interference at the site like this which would cause these issues your seeing.

Reactions:

(2)

Dennis Eaton

In reply to Michael Miller

IPVMU Certified

It is definitely worse at night when the scene becomes very complex. During the day there are people milling about and minimal movement from rides. At night the scene becomes extremely complex with thousands of visitors moving around, flags waving, large rides moving, and lights blinking. Originally I had thought it was a bandwith problem, but I’m at a loss. I believe the ubiquity radios operate at 3.6 GHz

Reactions:

Michael Miller

Can you post the model number and pics of the radio/antenna setups?

Reactions:

Dennis Eaton

In reply to Michael Miller

IPVMU Certified

Reactions:

Dennis Eaton

IPVMU Certified

Here’s one on the bench. Looks like it’s a NanoStation365.

Reactions:

Shannon Davis

IPVMU Certified

One question is how many FPS are the cameras set at? Are they still at factory maximum. An older computer viewing through a browser could be an issue as browser viewing also takes up a lot of resources. If you opened up the task manager I would bet the CPU is near maximum if you have all the cameras opened at once. The 6055's have zip stream built in. Make sure and adjust that properly.

Does the fair grounds have free WiFi? If so then you could potentially have 30,000 visitors trying to connect to it so their could be a huge amount of interference. Use a WiFi analyzer on your phone if you have android or feature you can do with Windows 10 and see which channels are being used and then switch yours to a different channel. That will help some.

If there are some obstructions with the line of site then make sure the units are on 2.4Ghz as this help the connection with the longer wave length.

So this year have a better PC for the system with you VMS and most likely you will have much fewer issues.

Reactions:

(1)

Michael Miller

If there are some obstructions with the line of site then make sure the units are on 2.4Ghz as this help the connection with the longer wave length.

I would not recommend using 2.4Ghz for backhaul for any setups like this. Way too much interference. Having built out WIFI network for a much larger festival then this I would strongly recommend using radios in the 50/60/70/80Ghz range.

Reactions:

(1)

Dennis Eaton

In reply to Michael Miller

IPVMU Certified

How strongly do you feel that could be playing a part in this issue?

Reactions:

Michael Miller

Any time you pack a lot of people in a space with mobile devices you're going to have interference issues to deal with. First, I would check is if your using the right radios and antennas then look at the settings of the radios and check for interference.

Reactions:

Dennis Eaton

IPVMU Certified

I hadn’t thought about the massive amount of interference with that many devices. I think we were on 3.6 GHz. If I remember right, Ubiquity uses an odd frequency in that range. I tried reducing frame rate, resolution, sharpness, I frames, and compression, but nothing caused the PTZs to remain responsive on a consistent basis. I will definitely use a high-quality CPU this year.

Reactions:

Shannon Davis

IPVMU Certified

The ultimate test would be if you could recreate the environment and try just one camera and see if you still have issues.

Reactions:

Dennis Eaton

IPVMU Certified

I think this is a once-a-year scene because the event rolls into town for ten days once a year. I can’t imagine a more complex environment, so it’s kind of a fix it as we go situation. I could disable all but one of them and add them one at a time. I wonder if it’s a network issue or if the CPU is struggling with processing codecs and PTZ controls under heavy bandwidth when it gets busy at night.

Reactions:

Undisclosed #1

IPVMU Certified

The ultimate test would be if you could recreate the environment and try just one camera and see if you still have issues.

If he still has the footage from last year and a couple or eight spot monitors handy, he could roughly recreate the complexity/activity of the scenes by pointing and zooming each PTZ at a monitor which is playing back that camera’s original footage.

Of course there won’t be any WiFi interference, but it would probably come pretty close to recreating the actual network and camera cpu load.

Maybe even use 2 4K monitors with quad matrix, depending how close together the PTZs are.

Reactions:

Dennis Eaton

IPVMU Certified

I think it’s worth a try and definitely easier while the cameras are on the ground

Reactions:

Chris Chambers

•Mar 14, 2019

I have seven Hikvsions viewing in IE11 now on a Core i9-9900K, which has 8 cores and 16 threads. One 3MP, three 4MP, and three 4K cameras, and they all appear to be running close to their max bit rate due to noisy low light. (And I always stick with max video settings across the board since I only record on infrequent events.) The CPU is at about 35%, but on my older i7-3770K with 4 cores and 8 threads four cameras would take the CPU much higher than that.

In the past I have seen numerous scenarios where an overloaded PC could be extremely unresponsive, such as Symantec's protection suite causing about a 30 second delay from when the mouse was moved to when the cursor actually moved onscreen. In this case I have no idea whether your radio interference would be more likely, but a quick check of CPU usage with all cams connected should be easy. In addition to setting high gain, shut off noise reduction if possible and increase shutter speed. The background noise at night with those settings will simulate all kinds of activity, as noted in probably more than one article/test here at IPVM.

Reactions:

Undisclosed #3

Sounds like my ex-gf after our first weekend in Vegas(OP: Title Response Only First Impression(Unresponsive in Complex Scenes)).

Reading your post now with one eye open and craft beer fumes.

Reactions:

(2)

Undisclosed #1

In reply to Undisclosed #3

IPVMU Certified

bored already?

Reactions:

(1)

Undisclosed #3

https://linux.slashdot.org/story/13/05/16/1727256/linux-is-an-obvious-choice-for-automating-the-beer-brewing-process-video

Oh the Undisclosed appreciation I feel right now.

The beer fumes are now....however a once epic video...

Reactions:

(2)

Undisclosed Integrator #4

Without a powerful IT/IoT diagnostic tool, you're basically just taking guesses at what's wrong.

I would recommend looking at implementing a diagnostic tool for all your deployments; especially if they involve challenging network topology like wireless.

With that, you'll know exactly what's wrong and how to fix it.

Reactions:

(1)

Christine Jackson

Talk to your IT group and see if they plan on running any fiber in that area in the near future. (That is if you use the space for other events)

Once we ran fiber here with our Fairgrounds, we were able to deliver a rock solid solution with cameras running back to a multi site VMS on couple of servers. Fiber/Cat 6 is the most stable. Wireless solutions most always come with a few issues, unfortunately. Also, I agree that the CPU could be causing trouble as well. Also, a Wi-fi analyzer/diagnostic tool is a a good idea.

Reactions:

(1)

Jeffrey Hinckley

In reply to Jeffrey Hinckley

I am curious where you got the radios (this is licensed frequency I believe). Are you using nanostation radios as clients (at the cameras ) and as AP (at the headend?). Are these M seriies radios? Do you have latest firmware?

The nanostation M series do not have a great deal of resources to function as AP for multiple devices, and if you are using one as the central AP, they are directional in nature. Try using a Rocket AC Prism with omni at the headend and connect the cameras and control center by nanostation, nanobeam, or litebeam AC radios (via 5.8 GHz). We have used this setup in parking lots with 10-20 cameras (some PTZ) with no latency on a VMS. Also have this setup in a city with sectors in windows of a bell tower (about 100’ up) feeding PTZ cameras up to 2500’ away and several remote parking garage camera clusters with little problem (and quite a bit of noise in this band).

Can not help you if you are using browser based monitoring (which can cause latency).

Reactions:

Dennis Eaton

•Feb 23, 2019

IPVMU Certified

Yes, we are setting them to ‘Station’ at the cameras and AP at the headend. I do believe they are M series and they are all line of sight. The one with the most problems is about 125 feet away and has zero visual obstructions. The fairgrounds have been extremely difficult relative to cellular connectivity even though the entire county has very good connectivity with all of the major carriers. I’m really hoping to dial in the wireless before we launch our mobile program...but I am willing to cable it if we can’t figure it out. I’m going to start with a new CPU and an Omni-directional AP and work through each of your solutions until we get it right. I’ll post an update once the weather breaks and we get the system up and running.

Reactions:

Undisclosed #3

I would test.

Camera > RTSP> Local Lan Laptop.

Turn off multicasting in Axis 'Plain Config'.

Lets use VLC, axis-media/media.amp or find your URL here: URL:

So, from CAMERA, to SWITCH(bridge) to CLIENT you are deducing latency, connectivity, performance.

If you can, mirror the Axis camera port and separately the client port. Use Wireshark to digest the packet structure and sequence (easy to google),

Start up testing revolves around your camera output(OSI layers) and switch brand (configurations) and final destinations (NVR storage, VMS clients).

Accomplish this on the bench and your company is ready to explore deployment and service of such designed infrastructure.

Without understanding, everyone is using an assumption/reaction. You do not want this.

Integrator deployment sequence to production needs to be graphed and discussed here on IPVM.

Ciao!

Reactions:

(1)

Undisclosed #1

In reply to Undisclosed #3

IPVMU Certified

Lets use VLC, axis-media/media.amp or find your URL here: URL:

How well do the PTZ controls work in VLC?

Reactions:

Dennis Eaton

IPVMU Certified

I haven’t tried VLC yet. Due to the extreme complexity of the scene, I thought I would have to wait until the event to re-create the scene and work out the bugs. I can see there might be some options on the bench first. I can say that the problem only manifests once the rides are moving, lights are flashing, and there is a large crowd, etc...

Reactions:

Undisclosed #1

IPVMU Certified

I haven’t tried VLC yet.

To be sure, I wasn’t suggesting you do, as it might be of limited use since it it a player, not a PTZ controller.

Wireshark may come in handy at some point, it collects and analyzes packets. For instance you could capture the packet stream related to the control of the ptz, and then inspect to see what the timing of the individual packets are.

Reactions:

(1)

Dennis Eaton

In reply to Undisclosed #3

IPVMU Certified

Thank you for the detailed road map. Once I get all of the code from Wireshark, what am I looking for , specifically?

Reactions:

Gary Gibson

•Feb 28, 2019

I would try operating the cameras in "playback" mode if you still have the original SD cards containing last year's complex scenes. (Assuming your cameras recorded to their own SD cards). Do this during the daytime when there's little potential radio interference. Try it with one camera then start adding cameras to see at what level failure occurs. If there's no failures then radio interference is likely the culprit.

Reactions:

(1)

Dennis Eaton

IPVMU Certified

Great idea! The playback should recreate the bandwidth. Radio interference is a non-issue when the fairgrounds are closed. In theory, this should isolate the problem.

Reactions:

John Richardson

You can simulate complex scenes by temporarily setting a very high minimum gain in each camera. This will cause a grainy image that requires significantly higher bandwidth - similar to that of busy or night time picture. Apply this to all cameras and see if you can replicate the issue before changing too many other variables.

I would suspect that your problem is due to inadequate bandwidth. You are probably also using TCP transmission which can hide the visual artefacts typical of H.264 transmission problems by retransmitting lost packets. Tell-tale signs of this are often jittery video which seems to pause and speed up, combined with high latency.

You can start by setting an MBR bandwidth limit on your cameras then ensuring you get the most out of your wireless setup either due to settings / alignment etc.

The Nanostation 365's are not great performers and my choice would always be an AC 5Ghz setup - but they should be ample if configured properly. Be careful of situations where multiple cameras are connected via a 100mb port such as a field switch with multiple cameras connected to your nanostation. You might need significantly lower bandwidth limits in these situations than you might expect.

Reactions:

(1)

Undisclosed #1

In reply to John Richardson

IPVMU Certified

You can simulate complex scenes by temporarily setting a very high minimum gain in each camera.

Not a very low maximum gain?

Reactions:

John Richardson