Recording Stops For 15 Minutes Every Night, Why?

Dear All,

I am having issue at one of our customer site. We have 50 odds IP cameras on one server. Every night at 12:00am to 12:15am we are getting recording breakage (max 1 min)for randomly on some cameras(5 to 6 cam average).

Server has window server 2008 R2 and camera type is Sony.I am not doubting VMS as simillar setup on different server doesnt have this issue.

I have collected log. CPU/RAM behaviour is normal at night time.

Can anybody help me to debugg this issue?


Sounds like something is rebooting during that time period. Can you tell / check what if any equipment is going down during that time? Is it the cameras, the switches, the server?

Recording breakage is sometimes 8 to 10 seconds so camera rebooting is not possible as if it reboots than min breakge would be more than 1 min. Switches need to be check as there is multiple swithces involved(Access Switch, Distribution switch and core switch). But recording breakage happen on different cameras everytime with their switch rout,there is no pattern as such.

And it only happens between 12:00am to 12:15am? No other times during the day or night?

Yes it only happen between 12:00am to 12:15am(some times till 12:25).

Day time also have breakage but not everyday, it is like 3 - 4 days one time.

Is the network shared with other devices ? We had a similar problem with other network equipment and in the end it happend to be an old printer server who went to broadcast around a set time to find other printers. Which cause our network devices to lose connected for a few seconds.

Which VMS is it? OnSSI/Milestone by chance?

No. Its not Milestone/OnSSI VMS. Have you come across similar issue with Milestone VMS? what have you tried to solve?

Is it Exacq? How many guesses do we get?

Like others have mentioned later in the thread, the way Milestone/OnSSI archives video between databases can sometimes disrupt recording during the operation (typically scheduled around midnight), especially if the system storage drives are near full capacity.

It is more a specific potential issue to those VMSes, and others are not vulnerable to the issue.

Possible the recording software is doing some sort of 'midnight cleanup' for archiving of video? Probably not the case if it's happening any other time outside of midnight though...

If it's OnSSI, it's most likely the archive process moving the current database to the archive.

This happens once a day minimum, if archiving is enabled. You can change the time it happens. Also happens if you exceed any channel limits for the day. Database goes down while it deletes enough records to continue. This definetly applies to NetDVR 6.0, don't know if they changed it since.

Seems to me there's an electrical problem here. A solution that worked for me before: install a basic $130.00 APC UPS battery with it's bundled software (for isolating the NVR from the customer's own centralized UPS system) and it will record all instances of 110V power failure, outages & any undervoltages during the night hours so that you can go back next morning to see it displayed in a recorded log and show the customer. Besides, the software will safely shutdown your NVR and (given the BIOS has the "Power ON After Fail" option enabled) it will turn on again upon sensing 110V in the power supply.

D, op says:

"But recording breakage happen on different cameras everytime with their switch rout, there is no pattern as such."

So it seems either camera side power or a vms side software (not power) problem, yes/no?

John,

There is no power problem at camera side as well. VMS is writing video data on NAS. NAS is connected through Core switch. I have collected window perf log last night. I find following warning.

"High rate of 29 split I/O per second was detected. This represents 50 percent of total I/O processes. Consider size of I/O processes compared to disk format size and defragment or reformat any disks with a high split I/O rate."

Does anything releated to this warning?any idea?

VMS is doing midnight cleanup. That may hint that There may be VMS issue but as i said earlier simillar setup at different site working fine.

Can you change the cleanup time to be a couple hours later? That should eliminate or incriminate the VMS...

Intermittent connectivity in different ports = a sign of a dying PoE switch, maybe ?? .. It would be helpful to see for how long the switch has been deployed and operating non-stop. If 5-10 years have passed, maybe it's nearing it's end-of-useful life.