Subscriber Discussion

Ubiquiti Network Goes Offline

JH
Jay Hobdy
May 03, 2017
IPVMU Certified

We have a client who specifies Ubiquiti cameras, switches, radios, etc. They have 15+ properties set up without issue.

 

We just installed a 30 camera project with 10 wireless links

 

Head end:
Server
Unifi 24 port switch
UVC-G3 and UVC cameras
Nano Beam AC 16dbi radios NBE-16-AC
600VA rack UPS

 

Outlying buildings:
TS-8 switch
350VA UPS
UVC-G3 cameras
Nano Beam AC 16dbi radios NBE-16-AC

 

switches and radios have ping watch dog enabled. So they reboot if they go offline

Radios are set to auto select frequency

 

The issue is groups of cameras go offline for 30-60 seconds every few minutes. these groups are buildings so it mat be 1-2 buildings at a time. This part of the property is like a Y where 2-3 links come to a common point, then everything goes back to the office across one link. Network traffic is around 50 Mbps at the "junction". We loose cameras at all points, even at the ends of the link with the lowest bandwidth.

 

Is there any way to log exactly what is going offline? We work with the clients IT department and they know this stuff pretty well but it seems like they just want us to start replacing switches.

 

Can the auto frequency be an issue? If the radios are changing frequency will they drop offline momentarily?

 

I can upload a site map if it helps

SN
Simon Nazaretian
May 04, 2017

I would start by launching multiple command prompts from your server and begin to continuously ping the management IP of each device starting from your core switch, to the following switch / AP, to the hosts (cameras), etc. 

Assuming a windows PC and a class C address "ping 192.168.x.x -l 1 -t". Arrange the windows so that you can see them all at ounce.

You should be pinging at least 1 x AP on each "leg" + 1 IP camera. 

This should show you what device along the way is dropping packets. 

Also, what does the ubiquity management software telling you about the APs? Have you checked logs?

(1)
(1)
JH
Jay Hobdy
May 04, 2017
IPVMU Certified

Do you mean at one time just ping everything related to the building? Like AP, Station switch and a camera? Run that for 15 minutes or so, then move to next building?

 

We do not have access to the management software, just the devices. Client controls all software. I saw a log on one radio but it was not helpful.

 

I was poking around in a radio just now and the radio went offline. I had to wait for it  to reconnect.

SN
Simon Nazaretian
May 04, 2017

Jay what I meant was that you should be pinging every device that is involved in the transport of packets from your cameras to the server. 

For example, let's just say your server is plugged into switch 1 in building 1 which connects to AP 1. AP 1 creates a link to AP 2 that is plugged into switch 2 in building 2, and AP 3 plugged into switch 3 on building 3. 

I would have 6 command prompts open each one pinging a separate device. Windows 1 pings switch 1, window 2 AP 1, window 3 switch 2, window 4 AP 2, etc. 

This way you can see exactly which device stops receiving and responding to your ICMP requests. This will tell you which device is "going down" or at least will provide you with somewhere to start troubleshooting from. I assume here that your devices have been setup to respond to / allow ICMP requests. 

For example, are both links that connect to your backhaul AP going down simultaneously? If so, then Jeffery could be spot on. If only one goes down he may still be right but now you know you have to troubleshoot only 1 AP and not two. If you loose connectivity to your AP doing the backhaul then it may, in fact, be something in your switching configurations, etc.

Really hard to tell without more information.

As far as the client controlling the network and not giving you access to the management console I consider that asking you to troubleshoot with a blindfold on. Possible, just harder. If you are on good terms with the IT department, as that they spend a little time with you and show you why they think it is the switches.

(1)
Avatar
Jeffrey Hinckley
May 04, 2017

This could absolutely be because of the auto frequency.  Also, if you are in the 5.3 or 5.5-5.7 band, you could have a dropout due to dfs.  I recommend using a frequency scan, and select frequencies to use in the 5.8 band.  Reduce the channels on the outskirts to 20 mhz and the primary backhauls to 40 mhz.  plot these out on a table so you do not have any interference.  You are using the 16dbi models, which have a wide beam width, plus these radios are known for problems.  Use the 19 dbi units.  If the links come back to a common point, or are multipoint, try using point to point instead.  If these are all in auto frequency, and have 40+mhz channels, that is probably the culprit.  Use the rocket prism for multpoint and enable transparent mode for client and access point radios.  Consider the directional lite beam.  Consider the new isobeam with rf isolator for sites with multiple radios or places prone to interference from the backside.  Put a 10 mhz channel space between each channel, since these are not using real filters.  For example, if you are using channel 5800/20 (5790-5810), use channels 5770/20 or 5830/20 for this seperation.  You can use 5.3 channels if needed but only for short leg end of the line links.  Never use auto frequency.

Nanobeam 16 radios should only be used for short links away from potential interference (parking lots).  I have used these a few times, and they have been horrible, even compared to the nanostation 19.  Probably has an inferior processor or hardware board.  

I miss these technical discussions like the old days, which have been replaced by sensationalism reporting and finger pointing.  Probably due to the Trump era.

(1)
(3)
(1)
DR
Drew Rollins
May 04, 2017

This is a great place to start. I never use auto-frequency because it can change and cause a drop out while the connection re-establishes. The logs in the radio will tell you if it is having to re-negotiate. I'd also hop on the Ubiquiti forum, there are some know-it-all jerks on there but you will get some help. 

(1)
Avatar
Jon Dillabaugh
May 04, 2017
Pro Focus LLC

Jay, quick question. Are you using a dedicated NanoBeam AC at the head end for each remote link? Are you trying to use a single NanoBeam AC as a PtMP at the head end?

Reason why I ask is because it seems as if they recommend using a LiteBeam AC for PtMP links.

Another issue may be interference between multiple NanoBeam AC units at the headend. If they are too close and aren't shielded from each other, that may be an issue as well.

(1)
JH
Jay Hobdy
May 04, 2017
IPVMU Certified

 

Apartment community

 

Head end is near bottom of map.

 

All dashes are wireless links

 

All links are PTP

 

16dbi is all we could get. Seems like 19dbi is on eternal back order

 

we have frequency set to auto per clients instructions, and we are using an 80Mhz channel width.

 

I think the first thing to do is set frequencies and narrow channel width.

 

Since everything is Ubiquiti (cameras, wireless, switches) which software would I ask for?

We use Ubiquiti radios, usually Loco's and rarely have issues, but we only do a few cameras per link. We never used the management software

Avatar
Jon Dillabaugh
May 04, 2017
Pro Focus LLC

If you buy through a decent distributor, you can give them a call to get their take, but I would agree with much narrower channel widths. Auto frequency would likely be OK with narrower channels, more channels to choose from.

Another question I have, did you create unique wireless networks for each set of links? Are they all using the same SSID and key, or are each link unique? Did you lock the stations to the appropriate AP? Maybe the stations are AP hopping or creating loops?

I am wondering if you could have installed a few of the LiteBeams instead of jumping from building to building. This would greatly cut down on the number of hops from end to end. I don't know about elevation or LOS issues from the simple map provided, so this may have not been possible. BX10 is five links deep.

(1)
JH
Jay Hobdy
May 04, 2017
IPVMU Certified

We buy from Doxxxx Raxxxx. They pretty much specialize in wireless I believe.

 

We do not lock the links as i believe that locks the station to ap via mac address. If the ap is replaced you need to reassociate the station.

 

We do create a unique ssid for each link. Link 1 is SB1, link 2 is SB2. We associate the station with the AP.

 

I do not think the links are hopping.

 

Not sure how the lite beams work but there was no line of site over the buildings. We were concerned about the bandwidth and multiple links but client was confident in it.

Avatar
Jon Dillabaugh
May 04, 2017
Pro Focus LLC

I would give your dist a call then. Sounds like you have most things covered, other than the 80 Mhz wide channels.

UI
Undisclosed Integrator #1
May 04, 2017

Definitely ditch 80 MHz channels - with that you only have 2 open channels to use without the use of DFS.

(1)
MM
Michael Miller
May 04, 2017

We just did a similar install but with Avigilon cameras for a car dealer.   Used Rockets for APs and NanoBeams for the Stations. You can see the layout below.  

We are running OS8, DFS channels and 20mhz channel width.  22 total Avigilon cameras from 3MP to 4K.  Also there are about 35 other SSIDs in the area so lots of interference.

I have seen some issue with the NanoBeams that the radio works and the connect to the AP but they don't pass any traffic until we reboot them.  Otherwise everything works well for the amount of traffic the network is handling. 

 

(1)
MD
Matthew Davis
May 04, 2017

Off topic question Jay. What software did you use to draw that map? Thanks

JH
Jay Hobdy
May 13, 2017
IPVMU Certified

conceptdraw pro

DL
DC Long
May 04, 2017

Install at least one Mikrotik router at strategic point(s) and you can use the tools and logging of Router OS to see into the network. 

(1)
DL
DC Long
May 04, 2017

If you use one of these

https://routerboard.com/RB750Gr3

you can install the Dude Network Monitoring System and live map the Network 

to any windoze PC.  Setup the  SNMP and you can monitor and log link speeds, breaks, etc.  for any vendors products. 

(1)
DW
Dennis Widdows
May 04, 2017

Just an aside...Some of the Nanos had a PORT flapping issue which mirrors what you are describing...I have never had this issue in the field my self however if you go over  to the Ubiquiti  forums and do some looking you will see that this has been an issue from  last year...check the Manf date on your Nanos...and ask your questions on that forum...

 

Good luck

 

 

(1)
DW
Dennis Widdows
May 04, 2017

Oh and Auto Freq and 80 mhz wide channels are No nos...

 

Best

(1)
JH
Jay Hobdy
May 13, 2017
IPVMU Certified

So the saga continues

 

We installed a 2nd location for the same client, and same issue. Cameras keep going offline.

 

1st site I installed Air Control and tried to monitor the radios but it seems like a port issue on port 9081 and the client did not respond to my request to investigate.

Double Radius, my distributor was of absolutely no help. Their 2 options were contact Ubiquiti or take our class.

 

Ubiquiti was worthless for support. After a 90 minute online chat session, the only solution was upgrade firmware.

 

We have not been back to the 1st site since are still trying to figure out how to troubleshoot it.

 

We had a 2nd site already started and we just finished it. Same issue. From the head end I could see all the cameras drop, except one that was connected to the head end. I narrowed channel width to 30Mhz on all major links and used specific frequencies (no auto frequency) and same thing. But they do come back up much quicker. They drop off for 5 seconds or so, then pop back up.

 We did upgrade firmware to the newest one 8.02 I think, at the 2nd site.

 

If I can get Air Control to monitor, will it tell me exactly why they are dropping off?

 

Here is a picture of new project with the assigned frequencies and channel widths. The AP and ST may seem backwards but we designed it so the AP would be on the office with the future head end. I am assuming it does not really matter as the devices communicate both ways?

 

right click and open in new tab for full size image

MM
Michael Miller
May 13, 2017

I have never had good luck with 30Mhz.  I would stick with 20 or 40Mhz.

Do you have any cameras that are connected to the NVR without wireless?   Are these showing offline too?

Is their WIFI in the building you are mounting the APs on?

 

I have been installing UBNT PTP and PTMP for many years and recently I have been having nothing but issues with the NanoBeam ACs and Rocket ACs.   Two installs we just did I have had 2 rockets fail and 2 Nanobeams within 1 week.  I also have NanoBeams stop passing traffic but still stay connected and you can access the web GUI.  Only a reboot fixes the issue.   I have setup Ping watchdog in the switches to monitor the head end AP so if pings fail it auto power cycles the AP.    I have NEVER had to do this before.

I am currently working on quoting Siklu to replace UBNT gear. 

 

 

(1)
MC
Marty Calhoun
May 13, 2017
IPVMU Certified

Change to the newer Unify, I have had better luck, and we always run on 10MHz as opposed to 20Mhz or 40Mhz that closes the path size considerably but harder to align, once locked up all the way around, never had a failure. I have 40-50 antennas on one site and all are functions 100%.

(1)
MM
Michael Miller
May 13, 2017

Unify is the WIFI line Airmax is the PTP PTMP gear.  I think you mean the latest Airmax gear.  I have hundreds of NanoStations in the field without issue but there seem to be issues with the new Nanobeams.  Also, you need to know how much bandwidth the link requires before you pick channel width and best practice is to use the smallest Channel Width needed for the bandwidth the link requires. 

(1)
(1)
JH
Jay Hobdy
May 13, 2017
IPVMU Certified

What is the difference between 20 and 40 Mhz widths? Capacity?

 

Is there any chart that shows the relationship between width and capacity?

 

 

Its an apartment community so I am sure there is a ton of personal wi-fi but nothing the community itself is providing.

 

One camera #25 on the map stays online when the rest go off.

 

 

MM
Michael Miller
May 13, 2017

Yes the wider the channel more capacity but you also open up to more interference and you have less none overlapping channels to work with.

Since your using all PTP links to backhaul the video how much traffic are you seeing/expecting on the last link at the headend?  Looks like AP A01?

 

If you have not signed up for the UBNT form I would recommend you do so as that will be your best support option.

(1)
MM
Michael Miller
May 13, 2017

Hopefully, this isn't the issue we are seeing at it looks like a hardware issue which will require replacing all the APs

 

 

(1)
JH
Jay Hobdy
May 14, 2017
IPVMU Certified

Michael I will post over in the UBNT forums but before we run out for Mothers Day breakfast here is what I grabbed from my first Station S/05 on building 5.

 

I also noticed on the switch the port status would go to blocking, then learning, then forwarding. Wait a few minutes and repeat. I turned off ping watchdog and same thing. Then I power cycled the port and the radio and rest of the network came up. for now

 

The date and time are wrong. I think I fixed it

 

 

 

 

 

Mar 27 14:07:20 system: Start
Mar 27 14:07:21 syslogd started: BusyBox v1.19.4
Mar 27 14:07:21 wireless: wifi0 Scan request completed
Mar 27 14:07:21 wireless: wifi0 Scan request completed
Mar 27 14:07:21 FileSystem: Start check...
Mar 27 14:07:21 init: Run: /bin/dropbear -F -d /etc/persistent/dropbear_dss_host_key -r /etc/persistent/dropbear_rsa_host_key -p 22
Mar 27 14:07:21 init: starting pid 683, tty '/dev/null': '/bin/dropbear -F -d /etc/persistent/dropbear_dss_host_key -r /etc/persistent/dropbear_rsa
Mar 27 14:07:21 wireless: wifi0 Scan request completed
Mar 27 14:07:22 wireless: wifi0 Scan request completed
Mar 27 14:07:22 init: Run: /bin/lighttpd -D -f /etc/lighttpd.conf
Mar 27 14:07:22 init: starting pid 686, tty '/dev/null': '/bin/lighttpd -D -f /etc/lighttpd.conf'
Mar 27 14:07:22 wireless: wifi0 Scan request completed
Mar 27 14:07:22 wireless: wifi0 Scan request completed
Mar 27 14:07:22 init: Run: /bin/pwdog -d 300 -p 300 -c 3 -m 300 192.168.55.157
Mar 27 14:07:22 init: starting pid 687, tty '/dev/null': '/bin/pwdog -d 300 -p 300 -c 3 -m 300 192.168.55.157'
Mar 27 14:07:22 init: Run: /bin/ubntspecd -t -a -i wifi1 -j airview1
Mar 27 14:07:22 init: starting pid 688, tty '/dev/null': '/bin/ubntspecd -t -a -i wifi1 -j airview1'
Mar 27 14:07:22 dropbear[683]: Not backgrounding
Mar 27 14:07:22 pwdog[687]: pwdog: do_now=0, initial_sleep=300, timeout=300, retry_count=3, low_mem=300 exec=`reboot`
Mar 27 14:07:22 wireless: wifi0 Scan request completed
Mar 27 14:07:22 init: Run: /bin/ulogger
Mar 27 14:07:22 init: starting pid 689, tty '/dev/null': '/bin/ulogger'
Mar 27 14:07:22 init: Run: /sbin/getty -L ttyS0 115200 vt100
Mar 27 14:07:23 wireless: wifi0 Scan request completed
Mar 27 14:07:23 wireless: wifi0 Scan request completed
Mar 27 14:07:23 wireless: ath0 Set Mode:Managed
Mar 27 14:07:27 wireless: ath0 Scan request completed
Mar 27 14:07:28 FileSystem: End check.
Mar 27 14:07:28 wireless: ath0 Set Mode:Managed
Mar 27 14:07:29 wireless: ath0 Set Frequency:5.74 GHz (Channel 148)
Mar 27 14:07:29 wireless: ath0 Sending auth to f0:9f:c2:b8:a6:d3. Status: Successful (0).
Mar 27 14:07:29 wireless: ath0 Received auth from f0:9f:c2:b8:a6:d3. Status: Successful (0).
Mar 27 14:07:29 wireless: ath0 Sending assoc_req to f0:9f:c2:b8:a6:d3.
Mar 27 14:07:29 wireless: ath0 Received assoc_resp from f0:9f:c2:b8:a6:d3. Status: Successful (0).
Mar 27 14:07:29 wireless: ath0 New Access Point/Cell address:F0:9F:C2:B8:A6:D3
Mar 27 14:07:30 zcip: config br0 169.254.225.63
Mar 27 14:08:41 httpd[860]: Password auth succeeded for 'ubnt' from 192.168.55.60
Mar 27 14:09:18 ulogger[689]: interface=eth0 link=DOWN
Mar 27 14:09:20 ulogger[689]: interface=eth0 link=UP speed=1000Mbps duplex=FULL
Mar 27 14:12:22 pwdog[687]: PING Watchdog is checking 192.168.55.157 (192.168.55.157).
Mar 27 14:13:54 ulogger[689]: interface=eth0 link=DOWN
Mar 27 14:13:56 ulogger[689]: interface=eth0 link=UP speed=1000Mbps duplex=FULL

JH
Jay Hobdy
Jun 03, 2017
IPVMU Certified

Just an update, we had to enable the logs, ntp etc on all the radios so we could get accurate error logs.

 

On one site alone, we have 14 out of 20 radios where the LAN port goes down (flapping). We have an advanced RMA in process for 15 radios.

(1)
AS
Andrew Somerville
Jun 05, 2017

Jay,

Ubiquiti have (finally) admitted there is a hardware issue with the Ethernet port on the Nanobeam ac products. The reason why they have been in very short supply over the last few months is Ubiquiti stopped producing them but didn't make any announcement to this effect.

They have now launched their gen2 ac products - but there is only a Nanobeam ac 19, no ac 16.

It's up to you to decide whether you want to take the risk on a latest generation product or decide to wait a few months before deploying in anger so that you can be sure all the major bugs are ironed out.

We tend to stick where possible to the previous generation non-ac Nanobeams which are tried and tested.

And i would reinforce the earlier recommendations:

1. Use narrow channels wherever possible (10MHz, 20MHz)

2. Do not use auto frequency - assign a small block of frequencies to each link

3. Ensure sufficient physical isolation between colocated radios to prevent interference and false DFS events

4. Do not use adjacent channels on colocated radios for the same reasons.

Andrew

(1)
EJ
Eythor Johannesson
Jun 05, 2017

I would start off by checking cabling and connections, is this happening in all kinds of weathers? 

I have been doing this since 2012 and 99% of the installs have worked perfectly- however the few that didn't where to do because of shorts or even bad mains on location (construction sites).

Hope you find your solution soon anyways because I know how frusterating this can get :-)

 

Regards

Eythor

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions