Problem With Arecont Cameras On Ubiquiti Wireless Links - Any Ideas?

Maybe someone has had a similar problem. I have (2) AV8185 cameras and a 2DF8223 camera on a pole. Etherwan switch. Link is 500' to another pole. Other radio set goes from there to college campus fiber (1000'). Signal levels -50s with no noise. Frequencies 5.8 band 40 Mhz channels, no overlap. Cameras end up on Exacqvison Virtual.

I also have, from same campus fiber spot, radio link to pole (two 2CD4332), link from there to pole (two 2CD4332), multipoint from there to another pole (two 2CD4332), and other multipoint (two Darkfighter fixed). Links are also clear line of site, 5.3/5.8, -40--50s signal level, no frequency overlap. Quite a load (would use Nanobeam nowadays).

Now the problem. All Hikvision cameras on these dual radio setups stream all frames through the campus fiber network. The arecont cameras, however, have broken frames, or go offline quite a bit. I have put in hundreds of wireless applications (in the past, several Areconts on Tranzeo radio solutions). The only thing I can think of is that a) campus network blocks arecont packet/frame structure or b) the Ubiquiti radios have trouble passing the packet/frames of the areconts. Possibly Airmax protocol or other settings. (Even though the PTZ camera on the Arecont links passes frames, it is sluggish, unless you disable the Arecont cameras).

If anybody has had similar problems or can possible shed some light on this, I would appreciate it.


It's a shot in the dark, but did you make sure to enable WDS Transparent Bridging in bridging settings?

It is enabled both ends. Any shots are appreciated.

#becauseArecont

The Arecont jokes have officially 'jumped the shark'...

Sir, please accept my sincere apology for my doubt...

#wasArecont

Did you try turning off feeds from all non-arecont cameras and seeing if that improves their performance? And what framerates are you trying to run? We had an odd case with one Sony camera and one Arecont camera on the same switch by themselves on one of the spokes leading to the core switch. Bandwidth should have been there, but when trying to run the Arecont at full frame rate it kept going up and down by 10 fps difference sometimes. Doing the reverse and running the Sony full frame had no problems. Only way to get the Arecont at full frame was totally disabling the stream from the Sony. It's like the Arecont couldn't fight for it's share of the pipe, even though there should have been room enough for both.

"The arecont cameras, however, have broken frames, or go offline quite a bit."

The AV8185 is a relatively old model, and that generation of cameras were especially bad bandwidth hogs (especially at night) and struggled to maintain their frame rate.

What network protocol are you using? Have you tried different ones? The A&E spec says "The camera shall support a minimum HTTP, RTSP, RTP over TCP, RTP over UDP and TFTP network protocols."

how are you powering the Arecont cameras?

15 fps. Tryed turning off others. Originally had all cameras coming back through one bachaul last leg. I isolated the are cont on its own link. I am not saying this could be something enabled on the college campus switch network but I have to eliminate wireless.

I was hoping someone out there would have compared these cameras and their payload via wireshark. It could be airmax- protocol.

I have run these cameras in a harbor on tranzeo- without issues 802.11a radios.

I do now recall, though, issue last year where I did 4 ptz in city in a line. Worked great. Nanostation. I had a av3105 which I dropped in at second tepeat. All ptz, hik 2de series, ran great but that are cont would not stay stable.

These are new av8185DN-HB cameras. They are powered by 24VAC power supplies 40VA. Running 15 fps (have tried different frame rate, including 1 fps JPEG, to no avail.) That is an interesting concept, though, to try RTSP in Exacq.

I know when Arecont came out with H.264 HD cameras, in 2008, the integration with Exacq was TFTP. This changed, though, to another plugin later, while the TFTP was still available. Not sure if the Exacq plugin may still use this but I assume it went to RTP since the TFTP was to send high frame rate JPEG (even though it was used for H.264). I have seen information that many Enterprise switches block this and it does not play friendly with subnets. It sure looks like for some reason, the areconts choke the Ubiquiti radios.

Also, it should be noted that I have a speed test radio1-radio2a-radio2b-radio3 of 80-100 Mbps. The Arecont cameras (surround) do this day or night. In the day, I observe, in Exacq, at 8, 10, or 15 fps frame size of 5-35 kB (max 5 mbps per camera, average about 2-3 Mbps) or probably 15-20 Mbps for both surround cameras plus the 2df8223 is running about 2-4 Mbps. (Remember also, that I am using an Etherwan EX32000 switch, 10/100, thought of this). I have increase compression alot, decreased resolution in half, gone jpeg, tryed different frame rates, only had one online, both online, PTZ offline, etc. I have even cranked up the BW of the PTZ (low compression, etc) and the other 8 cameras on the other link (headend radio plugs into same Cisco Gig switch as this). Hikvision cameras run flawlessly (unless I peg the BW of the radios) no matter what.

Something bogging down Arecont either in the radios or customer network. This site is 3 hours away, but my next step will be to travel there with a server, change the AP radios (on campus building) to some alternates I have going to the server (simulate without campus network), and see if the bottleneck is in the campus network. Note: Campus security does have two of these cameras elsewhere with no frame recording issues.

I am hoping someone will come up with a solution (I will try the RTSP solution for sure), especially the Wireshark gurus or the Arecont/Exacq engineers (or past engineers).

are you using a PoE hub/switch on the pole with all four cameras by chance? and is this in conjuction with individual power supplies for each camera?

The reason I ask is I have had this problem before with 24Vac/PoE/PoE plus cameras and auto sensing PoE unmanaged switches, especially PTZ's. al lot of times the switch will detect a PoE device and try to power it up even if it lacks the power to do so. this causes "weird" things to happen. Prime example I had 2 WV-SFV631L's on a PoE plus unmanaged switch that was tied into a another managed switch that had 802.af only. Well if I reset the cameras or they were powered off the would not power up becasue the managed switch was trying to power up the devices though the PoE plus hub. Also on another job i had some 1st generation PoE IP cameras, they pulled right at 14watts. (your arecont cams pull 9-11 Watts ish from what I could find) so if the cameras got really hot ( like in the summer) the would pull more power and trip the PoE midspans we had and go offline till they cooled down and then they would restart. also the imagess would start fail right before this happened.

not sure if this is your problem but I dont think it is protocol issue. as you said they work stand alone else where. not to mention the spec sheet says it supports RTSP and RTP.

Switch is not poe enabled. I have seen issue you are talking about. You would think manufacturers would disable poe if 24/120 is sensed. Same holds true for axis encoders. A tech mistakenly connected the included dc power, but it was plugged into poe power. Went back and forth until it was discovered.

For the arecont cameras, the frame loss and reconnect is in many cases far less than standard boot time if power was issue.

For more information, the switch used at the pole with the (2) Arecont, (1) 2CD8223 PTZ, and NSM5-US is an Etherwan EX420051A. PTZ powered by included Hi-POE injector. Arecont cameras heater/blower and camera power leads connected to 24VAC plugin power. (I did not provide POE injectors for cameras on project, but am waiting for response from installers whether these may have been used in addition to camera 24VAC power).

The PTZ runs fine, but becomes sluggish with the Arecont cameras enabled on Exacqvision.

http://us.etherwan.com/products/ex42000-series

I have never seen an unmanaged switch with QoS before, how does that even function? its says it desides priority for packet management based on what is being sent and how it "sees" packets that are passing though. it has some CoS values but with out knowing what you are passing its kinda meaningless. its possible the switch doesnt like what arecont is passing though packet wise and the auto QoS is telling it to wait?

there is a very wierd issue somewhere with how this is configured, you may have to replicate your setup on a bench or swap out different peices of equipment to narrow down where to give a closer look.

Thanks Ed, I will consider and look closer into the switch. Have not used it with these cameras, but have with various others with no issues.

Does anyone know of the RTSP address for AV8185DN that can be used with Exacq. We have tried all of those in the Arecont resources and used various leads per Google.

I think, but have never confirmed, that you add a string to the end of the RTSP stream for each imager.

Based on their API manual, I think it's rtsp://IPADDRESS/h264.sdp?ssn=X

X is the stream number, which I would hope is simply 1, 2, 3, and 4 in this case, but I don't know for sure.

You can actually generate that from, I just tried, with a AV3105 from the web interface page. Works fine in exacq. Tried that remotely with the 8185 and it did not work. I did notice though when in there (been a long time since I worked with Arecont, back to 2008-09) that you can enable MTU (1500). We noticed that the Etherwan switch supports 1582, I believe, but still, why would anybody put this on a camera??? Hmmm.

I have put together an office similation of this system (finally found a tech who kept one of these left over from one of my projects). Have to boat home now (in Maine, 5" of range, Portland has 4' of water on the streets).

Going through my old Arecont parts box, I did come across probably the first H.264 HD cameras ever made (really, these were probably the first off the assembly line). Arecont AV5105 (six of them) bought I believe June or July 2008. Kind of like finding an old quarter (they would say "POE only" on them, even though there are AC/DC terminals). These all died by 2010-2013. Warranty was 1 year.

Jeffery, I think that Eddie might be on to something regarding the Etherwan and its QoS implementation. As you may or may not know it uses 802.1p which is MAC level QoS.

802.1p is not backward compatible and can lead to instability on networks with non-802.1p switches. This is because older switches will misinterpret the header used by the 802.1p protocol. It is important that the switches, Ethernet cards, and device drivers are all 802.1p compatible. tech-faq

It is contained in a 4 byte extension to the Ethernet frame. I don't what device would be tagging the frames this way, but if they are tagged then a 1500 MTU in other devices might be to small.

Its a long shot, but it's worth looking at the size of the frames when wiresharking.

I asked our remote services department to use RTSP and see what enabling and setting MTU on Arecont to < 1500. Reply so far.

"So we can pull RTSP with minimal breaks in video on .159. This is the camera that has had the worst connection issues so far. I cannot even get a video stream to exacq now with the normal addition of this camera. I can pull .160 via rtsp and via the normal exacq connection reasonably well. I am going to now turn down the MTUs on the .159 camera and see what effect this has."

I apologize if this is ridiculous on some level, but have you tried replacing the Arecont with a Hikvision or some other camera? At some point in your testing the lost money from manhours spent is counter productive. Change the camera and if problem is solved, go make some money.

Related: Hikvision is soon to release an 8MP multi-imager camera. I am not saying this is the right solution here but in the future this should help as a low cost alternative to such Arecont issues.

Jeff,

my name is Cosimo Malesci and I am the Co-Founder and VP Sales at Fluidmesh. Arecont cameras are tricky as they are more sensitive to wireless transmission. You need a custom QoS to get them to work reliably on Exacq. We run them on our Ponte and Volo links with no issues. I suggest you consider swapping the radios if possible so you stop burning money trying to fix it. You are not going to be able to get the right QoS on Ubnt radios unless they send you a special firmware. Let me know what I can do to help.

Thank you,

Cosimo

cosimo.malesci@fluidmesh.com

Cosimo, why are Arecont cameraas more 'sensitive' to wireless transmission? What does Arecont do that other manufacturers don't?

John

sorry for my late reply I was traveling. I don't have specific details on the Arecont and Exacq implementations but based on our wireshark analysis the Arecont cameras disconnect as soon as some of the management packets between the camera and Exacq are dropped. This can happen often in a wireless link because of the higher than normal error rate.

We solved the problem by adding those managemant packets to the built in QoS on the Fluidmesh radio and increaseing their number of packet retransmissions to make sure they don't get dropped.

Jeffrey,

I will be happy to get you a pair of Volo radios to try and see if they solve the problem. Norris was one of the first accounts we signed up in Maine when we started Fluidmesh in 2005 so we would be happy to help!

Talk soon,

Cosimo

its not what it is sending per se, its what the QoS is seeing thats the problem. QoS just sets priorities for what passing though the switch. its not a bad thing its just normally something you can configure on a managed switch that has lots of different devices besides cameras pluged into it.

the problem with "auto QoS" is you cant configure it so(which i have never seen auto QoS) if you have 2 different types of cameras with different bitrates and and so on, it could mess with transmission to and from your devices.

It may not be just the arecont cameras it could be any kind of camera put on the switch even two of the same brand just different types of cameras.

There are some nonstandard ip packets in the Arecont solution, I think. So your question about traffic being blocked might relate. Also remember different vendors provide different levels of robustness in their network protocol implementations and a wireless environment offers more than the average number of bad packets, retried packets, packets out of order, etc. I agree with the "try a Hikvision camera, as a network test device" comment. Find a way to check the network health and see if there are any issue, even small ones (bird flies in front of beam once a day, all other devices recover...)

There are some nonstandard ip packets in the Arecont solution...

Non-standard in what way?

I second UD 1

what is "non standard ip packet" ?

As I recall at least the discovery tool used nonstandard packets to find cameras. My definition of "nonstandard" is "very difficult to decode with Wireshark". I believe it was a DHCP packet but with crazy values in some of the fields. Something like a wireless device that is doing it's best to deliver connectivity could have an issue with that. Cisco switches configured to defend against malware attack would simply shut down the network port in your face when they see odd packets. One would confirm if that's relevant by capturing a trace of the network traffic and checking the traces to see if they decode ok.

Other devices do this, it's not a sin invented by Arecont Vision. You can reset Korenix switches if you send them the proper "magic packet". No username/password required. The right way to do device discovery (after being invited!) is to use ZeroConf and UPNP so that networks can be prepared for (legitimate) device discovery operations. (This is why you vendors should not be shipping janky discovery tools. It may well come back to haunt you later, and there are plenty of legitimate alternatives.)

Cisco switches configured to defend against malware attack would simply shut down the network port in your face when they see odd packets.

We had an issue with the Arecont OMNI camera on a Cisco network and what you are describing sounds exactly what we were seeing. I brought the OMNI in to test before we sold one and everything worked fine on our HP network but one we installed the camera at the customers site (Cisco network) the camera would work for a couple of days then stop. We had to reboot the switch to get the camera working again. Every other camera on the network and same switch worked fine. So out of 150 cameras only the Arecont had the issue.

When I called Arecont's support they said we are not compaitible with Cisco switches. I almost fell over when I heard that.

Years ago we had a 5MP Arecont box camera over a UBNT PTMP wireless link with 2 other ACTi cameras. The Arecont always had issues streaming and disconnecting from the Exacq server. We switched from Exacq to Avigilon VMS and haven't had any issues since.

I think I know the answer, but did they mean that model of camera was not compatible, or that Arecont is not compatible? Seems like a pretty important thing to leave out considering Cisco has such a large market share.

I don't remember exactly and I don't think I asked. I was in shock when he told me.

AND WE RESET THE COUNTER. It has been

_zero_ Days since someone reports lame network support in Arecont Vision cameras.

Btw, this is an example where there should be more votes for good comments. I've marked a few informative already and ask you to do the same.

Jeffrey, what did Arecont tech support tell you? I am just curious if they had any insights and/or excuses.

Just out of curiosity, is buffering active in the Ubiquity nodes? Try deactivating buffering. I have some experience with Arecont on projects with wireless with Exacq. This solved a few problems.

Sorry it took me so long to get back to this post. I am at site today. It is obviously in the wireless equipment and not the customer network. Though they are running vmware server for the campus, I put a server in directly connected to the wireless radios. if you just enable the 8185 cameras, they record, but every couple of minutes leave a couple second gap. With these and the 9 Hikvision enabled, the 8185 stop connecting, but the Hikvision continue to purr along. It is my theory that the radios are straining under the stress of the 8185 cameras. I am going to try replacing radios with Nanobeam for the primary links. These have better processors and increased memory.

I will not go much further than this. I will probably look to another solution. Had anyone had success with the digital watchdod 3x1080 panoramics with Exacq?

i have not had much luck with arecontvision support in the past. One time, when 4 out of 12 cameras went bad shortly after completion of a project, their sales manager called to state that they had a less than one percent failure rate. I believe that you all on IPVM voted them as one of the worst service manufacturers.

Does not work on Cisco switches. Makes me more suspect that the same may be true on these radios.

The Digital Watchdog 3x1080 panoramic only outputs a single 1080p stream so that would be a big step down in resolution from what you have currently.

Have you contacted Arecont support in the past week or two? Yes, their support has been voted one of the worst but I'd still try to see what they have to say as a call does not take that long and might reveal some useful information.

Geez, I am one of those guys that never asks directions. Engineers are always like this, I guess. Only as a last resort, I have my pride.

So, update. I went to site today. As you recall, to recap, I have two multihop links to 9 Hikvision cameras and 2 Hikvision 8185 cameras. These go into customer Foundry Networks switch on a "camera" routed vlan, which end up, across campus, on two VMWare Exacqvision servers. The Hikvision go fine, the Arecont connections stall, freeze, etc.

R1_R2_R2_R3 (2 x 8185 + 8223 PTZ)

R1_R2 (2 x 4312) R2_R3 (2 x 4312) R3_R4 (2 x 4312) R3_R5 (2 x 6026)

Frequencies (all 40 MHz) 5780/5840 for R1-R2. All others 5550, 5320, 5660, 5270. Bandwidth used on the all Hikvision links to R1 are about (40 Mbps) no problems. Bandwidth used to Arecont and PTZ (daytime, VBR) about 6-10 Mbps. I will check tonight. I bet this goes to 20+ Mbps.

R1 radios are powered and connected to a Toughswitch, which uplinks to customer network. The question I have is whether the problem with the connection the the 8185 cameras is the radios or the customer network to get back to the video servers (Exacq virtual).

As an integrator, I am concerned about the solution before connecting to the customer switch. Today I put in a server at the closet location where the AP radios end up. I disabled all of the cameras from the customers network, and put on this server. I started without connecting to the customer network. Everything works. There is occasional 2 second drops on the Arecont cameras, but still, everything works. I did a Wirelshark (just areconts, and then all cameras). If I connect the customer network, the Hikvision still work, but the Areconts freeze up.

This has to be some broadcasting event or ?? on the customer network. I did a scan, and there are no duplicate IP addresses corresponding to the Arecont cameras. This is a VLAN setup for camera devices routed to the server network (I hate this, would rather have server on camera subnet, with routing done by client). I did Wireshark on local server setup on and off the customer network.

So, the system works. Handoff of the patch cable to the customer and connection to their network is irrelavant if it causes problems?? Do I charge for all of my time. I hate this for using customer administrated transport systems.

Ouch. Do you charge for you time? I think yes, but I don't know the particulars of the contract. If YOUR equipment works, and the customer specified the equipment you furnished, yes you charge. I understand it may rub you the wrong way, we all like to deliver a working system.

But if they asked for XXX, and you delivered XXX, hand him the cable and send the bill for you time to date. If he wants any further assistance, he/she needs to pay for it. Here is why I say it. You have connected the Hikvision to his network and they run fine. It is clearly the Areconts that are causing the issue. If he specified that camera, he got exactly what he asked for. The onus is on the person who specified the equipment.

If he asks for recommendations and/or help, offer it to him by all means. I have to say, I would recommend putting in some other brand. You sound extremely competent. This is not an installer issue. There is clearly a conflict between his network and his choice of cameras.

I am curious about one thing only. Did you try to install with just one Arecont? I know you tried both, but if you installed just one and it still freezes, then no doubt, it is an Arecont issue.

How much bandwidth is allocated to the streams, including aggregate "choke points"? Or, is it at least 200% of expected maximum throughput? (If a peak of 20mb/s is expected, is at least 40mb/s provided?)