Problem With Arecont Cameras On Ubiquiti Wireless Links - Any Ideas?
Maybe someone has had a similar problem. I have (2) AV8185 cameras and a 2DF8223 camera on a pole. Etherwan switch. Link is 500' to another pole. Other radio set goes from there to college campus fiber (1000'). Signal levels -50s with no noise. Frequencies 5.8 band 40 Mhz channels, no overlap. Cameras end up on Exacqvison Virtual.
I also have, from same campus fiber spot, radio link to pole (two 2CD4332), link from there to pole (two 2CD4332), multipoint from there to another pole (two 2CD4332), and other multipoint (two Darkfighter fixed). Links are also clear line of site, 5.3/5.8, -40--50s signal level, no frequency overlap. Quite a load (would use Nanobeam nowadays).
Now the problem. All Hikvision cameras on these dual radio setups stream all frames through the campus fiber network. The arecont cameras, however, have broken frames, or go offline quite a bit. I have put in hundreds of wireless applications (in the past, several Areconts on Tranzeo radio solutions). The only thing I can think of is that a) campus network blocks arecont packet/frame structure or b) the Ubiquiti radios have trouble passing the packet/frames of the areconts. Possibly Airmax protocol or other settings. (Even though the PTZ camera on the Arecont links passes frames, it is sluggish, unless you disable the Arecont cameras).
If anybody has had similar problems or can possible shed some light on this, I would appreciate it.
It's a shot in the dark, but did you make sure to enable WDS Transparent Bridging in bridging settings?
Did you try turning off feeds from all non-arecont cameras and seeing if that improves their performance? And what framerates are you trying to run? We had an odd case with one Sony camera and one Arecont camera on the same switch by themselves on one of the spokes leading to the core switch. Bandwidth should have been there, but when trying to run the Arecont at full frame rate it kept going up and down by 10 fps difference sometimes. Doing the reverse and running the Sony full frame had no problems. Only way to get the Arecont at full frame was totally disabling the stream from the Sony. It's like the Arecont couldn't fight for it's share of the pipe, even though there should have been room enough for both.
"The arecont cameras, however, have broken frames, or go offline quite a bit."
The AV8185 is a relatively old model, and that generation of cameras were especially bad bandwidth hogs (especially at night) and struggled to maintain their frame rate.
What network protocol are you using? Have you tried different ones? The A&E spec says "The camera shall support a minimum HTTP, RTSP, RTP over TCP, RTP over UDP and TFTP network protocols."
how are you powering the Arecont cameras?
15 fps. Tryed turning off others. Originally had all cameras coming back through one bachaul last leg. I isolated the are cont on its own link. I am not saying this could be something enabled on the college campus switch network but I have to eliminate wireless.
I was hoping someone out there would have compared these cameras and their payload via wireshark. It could be airmax- protocol.
I have run these cameras in a harbor on tranzeo- without issues 802.11a radios.
I do now recall, though, issue last year where I did 4 ptz in city in a line. Worked great. Nanostation. I had a av3105 which I dropped in at second tepeat. All ptz, hik 2de series, ran great but that are cont would not stay stable.
These are new av8185DN-HB cameras. They are powered by 24VAC power supplies 40VA. Running 15 fps (have tried different frame rate, including 1 fps JPEG, to no avail.) That is an interesting concept, though, to try RTSP in Exacq.
I know when Arecont came out with H.264 HD cameras, in 2008, the integration with Exacq was TFTP. This changed, though, to another plugin later, while the TFTP was still available. Not sure if the Exacq plugin may still use this but I assume it went to RTP since the TFTP was to send high frame rate JPEG (even though it was used for H.264). I have seen information that many Enterprise switches block this and it does not play friendly with subnets. It sure looks like for some reason, the areconts choke the Ubiquiti radios.
Also, it should be noted that I have a speed test radio1-radio2a-radio2b-radio3 of 80-100 Mbps. The Arecont cameras (surround) do this day or night. In the day, I observe, in Exacq, at 8, 10, or 15 fps frame size of 5-35 kB (max 5 mbps per camera, average about 2-3 Mbps) or probably 15-20 Mbps for both surround cameras plus the 2df8223 is running about 2-4 Mbps. (Remember also, that I am using an Etherwan EX32000 switch, 10/100, thought of this). I have increase compression alot, decreased resolution in half, gone jpeg, tryed different frame rates, only had one online, both online, PTZ offline, etc. I have even cranked up the BW of the PTZ (low compression, etc) and the other 8 cameras on the other link (headend radio plugs into same Cisco Gig switch as this). Hikvision cameras run flawlessly (unless I peg the BW of the radios) no matter what.
Something bogging down Arecont either in the radios or customer network. This site is 3 hours away, but my next step will be to travel there with a server, change the AP radios (on campus building) to some alternates I have going to the server (simulate without campus network), and see if the bottleneck is in the campus network. Note: Campus security does have two of these cameras elsewhere with no frame recording issues.
I am hoping someone will come up with a solution (I will try the RTSP solution for sure), especially the Wireshark gurus or the Arecont/Exacq engineers (or past engineers).
are you using a PoE hub/switch on the pole with all four cameras by chance? and is this in conjuction with individual power supplies for each camera?
The reason I ask is I have had this problem before with 24Vac/PoE/PoE plus cameras and auto sensing PoE unmanaged switches, especially PTZ's. al lot of times the switch will detect a PoE device and try to power it up even if it lacks the power to do so. this causes "weird" things to happen. Prime example I had 2 WV-SFV631L's on a PoE plus unmanaged switch that was tied into a another managed switch that had 802.af only. Well if I reset the cameras or they were powered off the would not power up becasue the managed switch was trying to power up the devices though the PoE plus hub. Also on another job i had some 1st generation PoE IP cameras, they pulled right at 14watts. (your arecont cams pull 9-11 Watts ish from what I could find) so if the cameras got really hot ( like in the summer) the would pull more power and trip the PoE midspans we had and go offline till they cooled down and then they would restart. also the imagess would start fail right before this happened.
not sure if this is your problem but I dont think it is protocol issue. as you said they work stand alone else where. not to mention the spec sheet says it supports RTSP and RTP.
For more information, the switch used at the pole with the (2) Arecont, (1) 2CD8223 PTZ, and NSM5-US is an Etherwan EX420051A. PTZ powered by included Hi-POE injector. Arecont cameras heater/blower and camera power leads connected to 24VAC plugin power. (I did not provide POE injectors for cameras on project, but am waiting for response from installers whether these may have been used in addition to camera 24VAC power).
The PTZ runs fine, but becomes sluggish with the Arecont cameras enabled on Exacqvision.
I have never seen an unmanaged switch with QoS before, how does that even function? its says it desides priority for packet management based on what is being sent and how it "sees" packets that are passing though. it has some CoS values but with out knowing what you are passing its kinda meaningless. its possible the switch doesnt like what arecont is passing though packet wise and the auto QoS is telling it to wait?
there is a very wierd issue somewhere with how this is configured, you may have to replicate your setup on a bench or swap out different peices of equipment to narrow down where to give a closer look.
Thanks Ed, I will consider and look closer into the switch. Have not used it with these cameras, but have with various others with no issues.
Does anyone know of the RTSP address for AV8185DN that can be used with Exacq. We have tried all of those in the Arecont resources and used various leads per Google.
You can actually generate that from, I just tried, with a AV3105 from the web interface page. Works fine in exacq. Tried that remotely with the 8185 and it did not work. I did notice though when in there (been a long time since I worked with Arecont, back to 2008-09) that you can enable MTU (1500). We noticed that the Etherwan switch supports 1582, I believe, but still, why would anybody put this on a camera??? Hmmm.
I have put together an office similation of this system (finally found a tech who kept one of these left over from one of my projects). Have to boat home now (in Maine, 5" of range, Portland has 4' of water on the streets).
Going through my old Arecont parts box, I did come across probably the first H.264 HD cameras ever made (really, these were probably the first off the assembly line). Arecont AV5105 (six of them) bought I believe June or July 2008. Kind of like finding an old quarter (they would say "POE only" on them, even though there are AC/DC terminals). These all died by 2010-2013. Warranty was 1 year.
Jeffery, I think that Eddie might be on to something regarding the Etherwan and its QoS implementation. As you may or may not know it uses 802.1p which is MAC level QoS.
802.1p is not backward compatible and can lead to instability on networks with non-802.1p switches. This is because older switches will misinterpret the header used by the 802.1p protocol. It is important that the switches, Ethernet cards, and device drivers are all 802.1p compatible. tech-faq
It is contained in a 4 byte extension to the Ethernet frame. I don't what device would be tagging the frames this way, but if they are tagged then a 1500 MTU in other devices might be to small.
Its a long shot, but it's worth looking at the size of the frames when wiresharking.
I asked our remote services department to use RTSP and see what enabling and setting MTU on Arecont to < 1500. Reply so far.
"So we can pull RTSP with minimal breaks in video on .159. This is the camera that has had the worst connection issues so far. I cannot even get a video stream to exacq now with the normal addition of this camera. I can pull .160 via rtsp and via the normal exacq connection reasonably well. I am going to now turn down the MTUs on the .159 camera and see what effect this has."
I apologize if this is ridiculous on some level, but have you tried replacing the Arecont with a Hikvision or some other camera? At some point in your testing the lost money from manhours spent is counter productive. Change the camera and if problem is solved, go make some money.
my name is Cosimo Malesci and I am the Co-Founder and VP Sales at Fluidmesh. Arecont cameras are tricky as they are more sensitive to wireless transmission. You need a custom QoS to get them to work reliably on Exacq. We run them on our Ponte and Volo links with no issues. I suggest you consider swapping the radios if possible so you stop burning money trying to fix it. You are not going to be able to get the right QoS on Ubnt radios unless they send you a special firmware. Let me know what I can do to help.
its not what it is sending per se, its what the QoS is seeing thats the problem. QoS just sets priorities for what passing though the switch. its not a bad thing its just normally something you can configure on a managed switch that has lots of different devices besides cameras pluged into it.
the problem with "auto QoS" is you cant configure it so(which i have never seen auto QoS) if you have 2 different types of cameras with different bitrates and and so on, it could mess with transmission to and from your devices.
It may not be just the arecont cameras it could be any kind of camera put on the switch even two of the same brand just different types of cameras.
There are some nonstandard ip packets in the Arecont solution, I think. So your question about traffic being blocked might relate. Also remember different vendors provide different levels of robustness in their network protocol implementations and a wireless environment offers more than the average number of bad packets, retried packets, packets out of order, etc. I agree with the "try a Hikvision camera, as a network test device" comment. Find a way to check the network health and see if there are any issue, even small ones (bird flies in front of beam once a day, all other devices recover...)
As I recall at least the discovery tool used nonstandard packets to find cameras. My definition of "nonstandard" is "very difficult to decode with Wireshark". I believe it was a DHCP packet but with crazy values in some of the fields. Something like a wireless device that is doing it's best to deliver connectivity could have an issue with that. Cisco switches configured to defend against malware attack would simply shut down the network port in your face when they see odd packets. One would confirm if that's relevant by capturing a trace of the network traffic and checking the traces to see if they decode ok.
Other devices do this, it's not a sin invented by Arecont Vision. You can reset Korenix switches if you send them the proper "magic packet". No username/password required. The right way to do device discovery (after being invited!) is to use ZeroConf and UPNP so that networks can be prepared for (legitimate) device discovery operations. (This is why you vendors should not be shipping janky discovery tools. It may well come back to haunt you later, and there are plenty of legitimate alternatives.)
Btw, this is an example where there should be more votes for good comments. I've marked a few informative already and ask you to do the same.
Jeffrey, what did Arecont tech support tell you? I am just curious if they had any insights and/or excuses.
Just out of curiosity, is buffering active in the Ubiquity nodes? Try deactivating buffering. I have some experience with Arecont on projects with wireless with Exacq. This solved a few problems.
Sorry it took me so long to get back to this post. I am at site today. It is obviously in the wireless equipment and not the customer network. Though they are running vmware server for the campus, I put a server in directly connected to the wireless radios. if you just enable the 8185 cameras, they record, but every couple of minutes leave a couple second gap. With these and the 9 Hikvision enabled, the 8185 stop connecting, but the Hikvision continue to purr along. It is my theory that the radios are straining under the stress of the 8185 cameras. I am going to try replacing radios with Nanobeam for the primary links. These have better processors and increased memory.
I will not go much further than this. I will probably look to another solution. Had anyone had success with the digital watchdod 3x1080 panoramics with Exacq?
i have not had much luck with arecontvision support in the past. One time, when 4 out of 12 cameras went bad shortly after completion of a project, their sales manager called to state that they had a less than one percent failure rate. I believe that you all on IPVM voted them as one of the worst service manufacturers.
Does not work on Cisco switches. Makes me more suspect that the same may be true on these radios.
Geez, I am one of those guys that never asks directions. Engineers are always like this, I guess. Only as a last resort, I have my pride.
So, update. I went to site today. As you recall, to recap, I have two multihop links to 9 Hikvision cameras and 2 Hikvision 8185 cameras. These go into customer Foundry Networks switch on a "camera" routed vlan, which end up, across campus, on two VMWare Exacqvision servers. The Hikvision go fine, the Arecont connections stall, freeze, etc.
R1_R2_R2_R3 (2 x 8185 + 8223 PTZ)
R1_R2 (2 x 4312) R2_R3 (2 x 4312) R3_R4 (2 x 4312) R3_R5 (2 x 6026)
Frequencies (all 40 MHz) 5780/5840 for R1-R2. All others 5550, 5320, 5660, 5270. Bandwidth used on the all Hikvision links to R1 are about (40 Mbps) no problems. Bandwidth used to Arecont and PTZ (daytime, VBR) about 6-10 Mbps. I will check tonight. I bet this goes to 20+ Mbps.
R1 radios are powered and connected to a Toughswitch, which uplinks to customer network. The question I have is whether the problem with the connection the the 8185 cameras is the radios or the customer network to get back to the video servers (Exacq virtual).
As an integrator, I am concerned about the solution before connecting to the customer switch. Today I put in a server at the closet location where the AP radios end up. I disabled all of the cameras from the customers network, and put on this server. I started without connecting to the customer network. Everything works. There is occasional 2 second drops on the Arecont cameras, but still, everything works. I did a Wirelshark (just areconts, and then all cameras). If I connect the customer network, the Hikvision still work, but the Areconts freeze up.
This has to be some broadcasting event or ?? on the customer network. I did a scan, and there are no duplicate IP addresses corresponding to the Arecont cameras. This is a VLAN setup for camera devices routed to the server network (I hate this, would rather have server on camera subnet, with routing done by client). I did Wireshark on local server setup on and off the customer network.
So, the system works. Handoff of the patch cable to the customer and connection to their network is irrelavant if it causes problems?? Do I charge for all of my time. I hate this for using customer administrated transport systems.
Ouch. Do you charge for you time? I think yes, but I don't know the particulars of the contract. If YOUR equipment works, and the customer specified the equipment you furnished, yes you charge. I understand it may rub you the wrong way, we all like to deliver a working system.
But if they asked for XXX, and you delivered XXX, hand him the cable and send the bill for you time to date. If he wants any further assistance, he/she needs to pay for it. Here is why I say it. You have connected the Hikvision to his network and they run fine. It is clearly the Areconts that are causing the issue. If he specified that camera, he got exactly what he asked for. The onus is on the person who specified the equipment.
If he asks for recommendations and/or help, offer it to him by all means. I have to say, I would recommend putting in some other brand. You sound extremely competent. This is not an installer issue. There is clearly a conflict between his network and his choice of cameras.
I am curious about one thing only. Did you try to install with just one Arecont? I know you tried both, but if you installed just one and it still freezes, then no doubt, it is an Arecont issue.
How much bandwidth is allocated to the streams, including aggregate "choke points"? Or, is it at least 200% of expected maximum throughput? (If a peak of 20mb/s is expected, is at least 40mb/s provided?)
Started by Alex Jason
|less than a minute by Alex Jason|
Started by Undisclosed End User #1
|less than a minute by Undisclosed End User #4|
Started by Charles Rollet
|21 minutes by Undisclosed #1|
Started by Brian Rhodes
|less than a minute by Undisclosed Integrator #3|
Started by Undisclosed End User #1
|2 minutes by Undisclosed Integrator #3|
Back to Top