Subscriber Discussion

Recording Stops For 15 Minutes Every Night, Why?

UM
Undisclosed Manufacturer #1
Mar 02, 2015

Dear All,

I am having issue at one of our customer site. We have 50 odds IP cameras on one server. Every night at 12:00am to 12:15am we are getting recording breakage (max 1 min)for randomly on some cameras(5 to 6 cam average).

Server has window server 2008 R2 and camera type is Sony.I am not doubting VMS as simillar setup on different server doesnt have this issue.

I have collected log. CPU/RAM behaviour is normal at night time.

Can anybody help me to debugg this issue?

JH
John Honovich
Mar 02, 2015
IPVM

Sounds like something is rebooting during that time period. Can you tell / check what if any equipment is going down during that time? Is it the cameras, the switches, the server?

UM
Undisclosed Manufacturer #1
Mar 02, 2015

Recording breakage is sometimes 8 to 10 seconds so camera rebooting is not possible as if it reboots than min breakge would be more than 1 min. Switches need to be check as there is multiple swithces involved(Access Switch, Distribution switch and core switch). But recording breakage happen on different cameras everytime with their switch rout,there is no pattern as such.

JH
John Honovich
Mar 02, 2015
IPVM

And it only happens between 12:00am to 12:15am? No other times during the day or night?

UM
Undisclosed Manufacturer #1
Mar 02, 2015

Yes it only happen between 12:00am to 12:15am(some times till 12:25).

Day time also have breakage but not everyday, it is like 3 - 4 days one time.

Rv
Rogier van der Heide
Mar 02, 2015

Is the network shared with other devices ? We had a similar problem with other network equipment and in the end it happend to be an old printer server who went to broadcast around a set time to find other printers. Which cause our network devices to lose connected for a few seconds.

(1)
Avatar
Brian Rhodes
Mar 02, 2015
IPVMU Certified

Which VMS is it? OnSSI/Milestone by chance?

UM
Undisclosed Manufacturer #1
Mar 04, 2015

No. Its not Milestone/OnSSI VMS. Have you come across similar issue with Milestone VMS? what have you tried to solve?

U
Undisclosed #3
Mar 04, 2015
IPVMU Certified

Is it Exacq? How many guesses do we get?

Avatar
Brian Rhodes
Mar 04, 2015
IPVMU Certified

Like others have mentioned later in the thread, the way Milestone/OnSSI archives video between databases can sometimes disrupt recording during the operation (typically scheduled around midnight), especially if the system storage drives are near full capacity.

It is more a specific potential issue to those VMSes, and others are not vulnerable to the issue.

UI
Undisclosed Integrator #2
Mar 02, 2015

Possible the recording software is doing some sort of 'midnight cleanup' for archiving of video? Probably not the case if it's happening any other time outside of midnight though...

U
Undisclosed #3
Mar 02, 2015
IPVMU Certified

If it's OnSSI, it's most likely the archive process moving the current database to the archive.

This happens once a day minimum, if archiving is enabled. You can change the time it happens. Also happens if you exceed any channel limits for the day. Database goes down while it deletes enough records to continue. This definetly applies to NetDVR 6.0, don't know if they changed it since.

(1)
UI
Undisclosed Integrator #4
Mar 03, 2015

Seems to me there's an electrical problem here. A solution that worked for me before: install a basic $130.00 APC UPS battery with it's bundled software (for isolating the NVR from the customer's own centralized UPS system) and it will record all instances of 110V power failure, outages & any undervoltages during the night hours so that you can go back next morning to see it displayed in a recorded log and show the customer. Besides, the software will safely shutdown your NVR and (given the BIOS has the "Power ON After Fail" option enabled) it will turn on again upon sensing 110V in the power supply.

(1)
JH
John Honovich
Mar 03, 2015
IPVM

D, op says:

"But recording breakage happen on different cameras everytime with their switch rout, there is no pattern as such."

So it seems either camera side power or a vms side software (not power) problem, yes/no?

UM
Undisclosed Manufacturer #1
Mar 03, 2015

John,

There is no power problem at camera side as well. VMS is writing video data on NAS. NAS is connected through Core switch. I have collected window perf log last night. I find following warning.

"High rate of 29 split I/O per second was detected. This represents 50 percent of total I/O processes. Consider size of I/O processes compared to disk format size and defragment or reformat any disks with a high split I/O rate."

Does anything releated to this warning?any idea?

VMS is doing midnight cleanup. That may hint that There may be VMS issue but as i said earlier simillar setup at different site working fine.

U
Undisclosed #3
Mar 03, 2015
IPVMU Certified

Can you change the cleanup time to be a couple hours later? That should eliminate or incriminate the VMS...

UI
Undisclosed Integrator #4
Mar 03, 2015

Intermittent connectivity in different ports = a sign of a dying PoE switch, maybe ?? .. It would be helpful to see for how long the switch has been deployed and operating non-stop. If 5-10 years have passed, maybe it's nearing it's end-of-useful life.

(1)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions