Subscriber Discussion

Weird Issue With Milestone, No Recordings, Possibly A Switch?

JH
Jay Hobdy
•Apr 16, 2019
IPVMU Certified

We have a server running Milestone and for some reason it stopped recording. We rebooted the server, then restarted all services. Still no timeline in playback. Today is the 16th and we have one camera that recorded on the 10th for 45 minutes.

 

Out of 21 cameras, only 5 are up in liveview.

 

The managed Ubiquiti switch was on a different network then the server, so we changed the server IP, and we tried to change the switch IP, and the switch froze up. We never actually touched the network settings. It froze up while switch pages/tabs. We rebooted it from the scan tool, and it never came back on the scan tool. We then changed the server back to the original IP address to match the cameras. We still have 5 cameras up in live view so the switch is working in some capacity. We are working remotely so we have to get a tech out to hard reboot the switch.

 

Is it possible live view is working, but due to the switch having the wrong IP address, it may not be passing ONVIF details over like motion detection? We did not change any settings and server side motion detection is not enabled. I thought of changing 1 or 2 cameras to server side motion to see if recording started but did not.

 

I feel like we have to get all the cameras online and resolve the switch issue before we address the lack of recording.

 

I have not reached out to Milestone because I do not think its an issue with Milestone, at least not yet. I wanted other opinions.

 

Thoughts?

SD
Shannon Davis
•Apr 16, 2019
IPVMU Certified

Did you happen to update firmware in the cameras? I have had this similar type of situation happen and the cameras either quit recording or now view-able at all in Milestone. Typically firmware doesn't cause an issue but sometimes it can. 

(1)
U
Undisclosed #1
•Apr 16, 2019

Can you ping all 21 of the cameras? Access the webpage of all 21 cameras or only the 5 that are alive accessible? What firmware is running the switch? is this and edge or unifi? The switch IP is for managing the switch and does not affect what is plugged in unless you roll with same net/subnet. Before rebooting a switch -> log into the web browser or preferably SSH and check the running state of the switch. The last person on the switch could have made changes without saving the running config, do a back up if needed. Restarting from the scan tool without knowing the state of the running and backup config has hosed many network techs regardless of what brand you are working on, especially with no core dump.

If this is an edge switch, using firmware 1.8+ with the dashboard you can easily reboot ports, poe and watch a ping drop on that said port(camera attached). 

Good Luck on the fix, seems like you have 3 areas to troubleshoot until this goes down in the books as case closed.

1. Camera(s) layer

2. Switch Layer

3. NVR layer

JH
Jay Hobdy
•Apr 16, 2019
IPVMU Certified

No firmware upgrades.

 

Its an edge switch and my tech told me rebooting it may have caused it to default back to another configuration as it happened on another switch. Problem was the switch was locked up and was not responding.

 

I did not think the IP of the switch mattered, just wanted to double check.

 

I did not think to ping the cameras but we believe the cameras do not have POE/power. Its a 48 port switch with 21 cameras so budget should not be an issue. My tech thinks the 5 working cameras are on one side of the ports, and the down cameras are on the other side. We experienced something similar with another edge switch and had to RMA it.

I am going to try to ping the cameras later.

 

U
Undisclosed #1
•Apr 16, 2019
In reply to Jay Hobdy

Instead of hard power cycle, you have the option to disconnect the cameras and have the tech hold the default reset button. Let the switch come back up then log in and see what firmware the switch restored to, that will at least hint where the switch was prior to the issues. Then take a look at a few of the cameras logs, if supported by the camera type you have, see if the issue on the cameras that stopped recording have any corresponding time date syslog errors regarding...server connection or power issues(switch problem). I know this add to the tasks you need to complete but a little exploration may help find the root cause if it is still not known at this point.

Good Luck!

DG
Donald Gordon
•Apr 16, 2019
IPVMU Certified

Jay,

Could be as simple as licensing the cameras.  Depending on the version of Milestone, you may only have a 30 day grace period.

Don

(2)
(1)
U
Undisclosed #1
•Apr 16, 2019
In reply to Donald Gordon

Nailed it!  :D

U
Undisclosed #1
•Apr 17, 2019
In reply to Undisclosed #1

....back to the drawing board for Jay Hobdy!

 

Good Luck!

JH
Jay Hobdy
•Apr 16, 2019
IPVMU Certified
In reply to Donald Gordon

definitely not a license issue. 

(1)
Avatar
Mike Rose
•Apr 16, 2019

Support package out of date 

 

make sure they are current 

(1)
(1)
(2)
UM
Undisclosed Manufacturer #2
•Apr 17, 2019
In reply to Mike Rose

I know you were being funny, but just to keep old wives tales from perpetuating - Care Plus provides access to software upgrades and is not related to support. Customer owns the licenses so even if it as out-of-date, the software does not quit working. Fake news! ;-)

(2)
(4)
UM
Undisclosed Manufacturer #3
•Apr 22, 2019

Yes, in the future a PING test would be the place to start with, and if you get a PING response then it is nothing to do with the IP addresses.

And as UND 1 said, the IP and Subnet of the switch does not affect camera/recorder communication unless it is doing some sort of Layer 3 routing, which a simple PING test would point out.

Anything fancy setup in the switch setup like VLANs?

This wouldn't happen to be an HP server, would it? I've heard of a similar issue with Milestone on an HP server and they kept having to reload the OS and Milestone and restore the configuration from a backup. They never did pin down exactly what the issue was. Maybe an odd HP specific driver the Milestone software did not like for some reason. 

It could be a problem with the switch, with the whole switch going out or just a bank of ports. But that is what a PING test is for, particularly a continuous PING with (for me as a general rule) at least 60 iterations to check for intermittent problems.

I have also seen people accidentally configure a gateway address on both the camera network NIC and the business network connection NIC which can cause the server to have communication issues with both networks. Or accidentally duplicate the IP address of the camera network NIC on another device and you have an IP conflict.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions