Time Drift On A Recorder Without Internet Access

I have a few questions from other experts regarding networks that don't allow internet access to the NVR or DVRand time.

Lets say that there is a client that will not allow ANY internet access to a DVR. I have a few concerns about time. I know that there is most likely not many lawyers on here so any feedback will be considered. If the time drifts on a DVR, could there be the possibility that the footage could be thrown out of court in the event of a robbery?

Second question, we have seen a particular brand of camera that holds its own time and the recorded footage isn't "marked" properly in the recorded footage. Does anyone out there use the DVR as a time server so that all cameras are sync'ed. Understanding that if the DVR can't reach a time server, the time will be off across the board but at least it will still be sync'ed.

If the time drifts on a DVR, could there be the possibility that the footage could be thrown out of court in the event of a robbery

Anything is *possible*, though the probability of that is pretty low unless the time is off by a massive amount (days).

There seems to be a lot of FUD about video being thrown out if anything about it isn't perfect. Think about all the other kinds of evidence admitted, rarely is any one piece of evidence completely 100% irrefutable. That's why a defense or prosecution attorney will generally have multiple pieces of evidence to present.

If anything is over-the-top pointless it's likely to be thrown out, but time skew on video (or 5fps, or h.264) is not something that commonly gets video thrown out.

Video that is blurry, shows no distinguishing value or has other flaws that inhibit the visual value are more likely to cause it to be inadmissable.

For your specific case, there are GPS network clocks that can act as an NTP server.

Background reference: NTP / Network Time Guide For Video Surveillance

If you are concerned about this, the best solution is the following:

Dedicated GPS Servers

In systems where accurate time is required but the surveillance system is not connected to the internet, dedicated GPS based time servers are used. These devices retrieve time from GPS satellites via antenna, either mounted to a window or external, and act as NTP/SNTP servers for the rest of the network.

Dedicated GPS time servers sell for about $800 USD online (Veracity Timenet) and up. Advanced servers (such as Spectracom and Meinberg) with extremely precise nanosecond accuracy, redundant external antennas, and other advanced features sell for easily 2-3 times this price, or more.

In the evening class of Class 6 of the IP Networking March 2015 course, a student mentioned a GPS server which was only about $125 which is much cheaper than the $800 unit mentioned in NTP / Network Time Guide For Video Surveillance. Unfortunately the recording of the evening class was corrupted and so I can't find the name of the unit. I remember Brian Rhodes mentioning he was making a note of the GPS server and hope he can post it here if he has that information as the price is attractive if the device works well.

Good recall Luke. One of the students mentioned this Time Machines GPS SNTP unit. I've not found it for $125, but usually between $275 - $325 USD.

Thank you very much for posting this information Brian. Even at the price you've found, it is attractive assuming it works reliably.

When I briefly looked at the various feedback on reseller sites for that unit, several users said directly pointing client devices to the Time Machines unit was a mistake, because it cannot handle more than 4 or 5 simultaneous connections.

Instead they recommended configuring an internal NTP server to the device and to synchronize all devices to that NTP server.

This may help explain the lower relative cost: The Time Machines unit is not a comparable 'server'; as much as it is a networked SNTP GPS clock.

Hi Brian, that advice may well be true. The specs suggest otherwise as one might expect:

Unit is capable of serving 135+ synchronizations per second. This provides support for over 120,000+ devices updating every 15 minutes on the network.

The cost is most likely due to the fact that is a SNTP server not a NTP server. This means that the GPS device primarily gets its time from one source (with a backup server specified), and then updates downstream devices from this time.

It does not employ slew rate, like ntp servers, which means that the time served will just 'jump' directly to the time it gets from its primary server every refresh interval, often 10 seconds.

A true ntp server will use multiple incoming sources, and then 'gradually' adjust time to all clients, to avoid any jerky time changes, (slew rate adjustments).

Understand that NTP and SNTP in this context refer only to the algorithm used to determine the time, not the actual data format used to deliver that time. NTP and SNTP are essentially the same 'on the wire'.

<edit: this reply was meant for Luke, not Brian>

Would a GPS receiver not inherently use as many satellites as it can "see" for all its data? After all, the timing differences between multiple signals is exactly how it determines geolocation. I mean, I suppose one could customize a receiver to just pull the time from the first satellite it sees and ignore all others, but why bother when even the most basic chip will try to grab as many birds as possible?

Even with multiple satellites, the difference in timing between them is generally on the order of a few milliseconds - when your main concern is not allowing multiple clients to differ by more than a few seconds for evidentiary purposes, a few milliseconds one way or the other isn't a big deal.

Would a GPS receiver not inherently use as many satellites as it can "see" for all its data?

You would think so, but it goes back to it only running SNTP server algorithims. So since SNTP does not do any graceful and gradual adjustments to the time served, opting just to 'tell it like it is' whenever polled, its probably better just to communicate with one stratum server.

Otherwise you increase the occurence of time running backwards, where the time server gives out an earlier (even if accurate), time than it just did. When you have only one NTP incoming time source you should always be getting a later time for each request. Internal drift aside, this in turn should allow a smoother (even if technically less precise) time to propogate to the clients of the GPS.

Another alternative to an external GPS clock or WWVB reciever is an external modem that calls the NIST time service. Of course you need an available phone line and may have long distance charges to Boulder, CO, but the hardware cost is less than $20 for a modem on eBay.

In our experence this is more of an issue when you have multiple NVRs or DVRS. When you only have one recording device the time may be off but the camera are normally still sync'd.

We have a customer which they had a case thrown out with there previous system which they had 16 DVRs and the time was not sync'd between them. They had a break-in and the video (camera on DVR A) showed the suspect stealing and at the same time (Camera on DVR D) he was sitting at the bar.

The system has since been replaced with NVRs which are time sync'd so they no longer have this issue.

I am not sure why this issue arises. Outgoing initiated internet traffic that is periodic in nature & does not hold a continuous session, posses virtually no risk as it will use a dynamic port on the firewall which will automatically close when the time check completes. This type of traffic is much different to traffic initiated from the outside which will require a port forward or VPN or the like to be able to reach the intended destination on the network.

Port forwards leave a permanent hole in the firewall that can be exploited by a hacker. But a port forward is not required for a DVR to access an external time server.

If they still insist on not allowing outgoing internet traffic, then an option is to set up a time server on a workstation or NAS that is allowed internet access & then refer the DVR to the workstation for time sync.

Another problem is if there ISN'T a network to speak of, or a severely restricted one.

The one client I mentioned above, the extent of their network was their office machine and paypoints linked together, and then a DSL connection over VPN to their WAN. Users on the office machine had to login to the machine first, then to the VPN; outside internet access was not even possible on these machines, they ONLY communicated with the WAN.

I think a big part of the reason they didn't want the DVRs connected to this network was that they didn't want unauthorized people to be able to access their systems. Controlling access would require adding the DVR to their domain, and I expect that's a can of worms the IT guys just didn't want to crack open.

We regularly brought up the benefits of time sync, and how simple it SHOULD be, even just to have the DVR sync to the office and paypoint systems, but nobody wanted to go there. Having several levels of bureaucracy to go through probably had something to do with it as well, combined with the fact that "it's just policy". Either way, we were utterly forbidden to plug the DVRs into their network.

The issue with overly restrictive networks can be the lack of sophisticated enough networking equipment, but can also be unmotivated IT staff. You can set some pretty restrictive rules so only the DVR can access specific known Internet time servers. It just takes effort, but and many times IT's response is, "We can't do that, it would compromise our security", even when they could, and it wouldn't. As an IT person, I apologize on behalf of my brethern's behavior.

As a former-IT, sometimes-still-IT person, apology accepted :)

But yes, I know a lot of times it's just a matter of nobody wanting the hassle. Consider that in a major national organization, too, that the levels of bureacracy and red tape that must be trudged through, not to mention all the changes to network policy that must be created, changed, and/or implemented, means nobody wants to take the risk of the fallout landing on them if anything goes wrong.... even if it's something completely unrelated and the only connection is coincidental timing.

Ah yes, unconnected systems and time sync... these were the bane of my existence for several years with one client in particular. Nobody ever cares enough about time sync until something happens and then they can't find the video they want because the clock never matches their paypoint systems (which DO have access to a time server). And a big fuss is made about it at the time, the problem is explained AGAIN, solutions are offered AGAIN, and by the time someone decides it's not worth the trouble or expense, nobody cares about it any more anyway. At least not until the next time. Same goes for cameras being down or a system beeping incessantly for weeks because a drive has failed - everyone just ignores it until they need some video, then OF COURSE the video they need is on the dead drive, or should have been captured by the dead camera.

There was one instance with a lottery fraud investigation, where the lottery people were concerned enough about accurate time sync (matching the video time to the time on the lottery terminal, so they could ID the person buying tickets with a stolen credit card) that they brought me in to spend time with their investigator - he'd print a ticket, bring it back to me, and we'd compare the timestamp on the ticket to the time on the DVR. Did this several times, in fact... then he had me write a two-page statement on why the time was wrong and how we determined the proper offset. If you can use a method like this to prove time offset, there should be no problem in court.

Of course, none of this is new to DVRs - part of our annual preventative maintenance on the ol' MUX-and-VCR systems was correcting the time on the MUX, which was always off by at least a couple hours from the previous year... and those didn't even have network time sync as an option. Most didn't know how to handle DST either.

As far as camera vs. DVR timestamp, we've always used the DVR-based one - if the camera has onscreen time display, I'll generally turn that off entirely. In the lottery example above, they originally called me in because the cameras were showing a timestamp in 1970 - they were IQs and the installer hadn't set the time or disabled the onscreen display, and the investigator kept fixating on that instead of the DVR's time display in the playback window.

Keep in mind that camera timestamps really only apply to IP cameras, so if you still have an analog or hybrid system, you still need the DVR's timestamp. As you move from analog to hybrid to all-IP, it just makes sense to keep that time source consistent, especially if you're still using the same VMS.

Time synch can be a big deal on any video system due to the distributed nature of many of the components. We tend to need to synchronize timestamps down to the second or subsecond for best performance. And in some extreme cases (as cited above) even to work with video.

Some shops take a pretty heavy handed attitude towards network security, that being there shouldn't be one (a network that is) but rather an isolated enclave of systems with no functionality outside of themselves. I believe they will hang on for many years just as analog has, and may in fact take some permanent form, but will become increasingly rare.

In the mean time, the local NTP server is the most straight forward and direct solution. Go ahead and propose they get one or live with the consequences.

Matt, I have had to go to court with the video on a few occasions. Typically if you can explain it to the Judge's satisfaction (laymen's terms) he will allow it. It all depends on the judge really and most have common sense. Long story short, the Coroner established a time of death as 6pm. An hour later the suspect is at the ATM with the victim's ATM card, but the time still says 6pm. The teller had not changed the time to allow for Daylight Savings Time. Yeah, you have to throw the teller under the bus, but the suspect needs to be in jail. There was other evidence that helped establish his guilt, but his face in the video, with her card, she's dead.

OK, that brings about a question on a particular brand. Axis cameras on an Exacq system send a timestamp with the video. For instance, an installer didn't do his do-dilligence and forgot to set the time in a camera. The TZ was still set for GMT. The NVR was set to CST-6. The footage they were looking for was recorded at noon but showed up at 6:00 pm when they searched for it. It had the 12:00 timestamp but didn't show up until 6:00pm. Weird part was, we found it by searching that day at 3:00.......It appeared that the footage was recorded in the future.

Has anyone else experienced this with Axis cameras on Exacq?

Hey Undisclosed. I've seen Exacq do this with other cameras (most often ACTi, Vivotek, and Hikvision), but not Axis. Exacq generally tries to push the server address as NTP server to some cameras when it's added, but I'm not sure if it does so with Axis or not. Regardless, the time zone has to be set separately.

Axis cameras running Exacq apparently have such an issue, known colloquially as the dreaded 'Osixrules' bug.