Subscriber Discussion

Time Sync Issues Exacq And Hikvision Cameras

UI
Undisclosed Integrator #1
Apr 02, 2015

I have been fighting this issue for a few days now with no avail. I have Hikvision DS-2CD4112FWD cameras that seem to drift in time over a period of a day. Once i connect the cameras in the exacq VMS and review the video they are perfectly sync'ed up, then after about 12 hours they are off by 2-5 seconds. I have configured the exacq vision to be the time server, i don't have an alternate NTP server configured, the camera shows the NVR as the time server. I have modified the NTP update field in the cameras to update every hour, and they are still off. i found a windows task to sync the time, and set that to hourly with out any success. cameras are on a flat network on my testing bench.

Does any one have any suggestions or had any advice on this one? I have never had issues with cameras drifting this much in the past. Cameras are on the latest firmware, exacq is on the latest release.

Thank you

Avatar
Ethan Ace
Apr 02, 2015

I haven't seen any issues with Hikvision cameras drifting during our testing, but we run a dedicated time server on all the Exacq machines. I've actually never tried syncing to an Exacq server without it, now that I think of it.

Can you try installing that time server (it's free) on the Exacq machine just to see if it makes a difference?

Be sure to set "Allow other computers to sync to this computer" and "Always provide time" options in the setup page, here:

UI
Undisclosed Integrator #1
Apr 03, 2015

So i installed this, are you guys setting this address up in the IP Camera Time Server field as well?

so time server i have 127.0.0.1

and IP CAMERRA TIME SERVER: is blank since NTP is running on the exacq box there should be no reason to bypass it correct?

just double checking how you guys have been usng this program since i have never had a need for this in the past.

UI
Undisclosed Integrator #2
Oct 07, 2016

Did this ever solved? I am having the same issues. It is the exacq overlay time sync that goes way out? Searching video is a mess.

UI
Undisclosed Integrator #1
Oct 07, 2016

I still have the issue, I have tried everything, I even set the time sync interval in the camera to sync every 90 -120 seconds and they still drift. Its very annoying.

UI
Undisclosed Integrator #2
Oct 07, 2016

Did you check the internal time on the cameras? Mine are fine and in sync (so the time being held in the cameras are fine). Its the overlay exacq time that is off and when you review video the searches are way off. I have this issue on a small exacq server and Large site as well. I have been working with Exacq on this with no luck. If i restart the server everything is in sync for 2 days at least! I am beyond frustrated.

UM
Undisclosed Manufacturer #3
Oct 07, 2016

Just curious, why would you leave the time onscreen on the cameras ?

UI
Undisclosed Integrator #2
Oct 07, 2016

Undisclosed # 3 - What on screen time are you referring to? OSD on the camera or overlay on the of the exacq?

If you are referring to the camera OSD. I didn't leave it on. I turned it on... Its purpose is to help identify what the time is on the physical camera to see what portion of the system is loosing time.

To be clear it is the overlay coming from exacq that is way off. And when you search 2 cameras in sequence to follow a subject the system displays out of sync recordings.

JH
John Honovich
Oct 10, 2016
IPVM

FYI, Exacq emailed us saying that they saw this thread and are investigating the issue.

UI
Undisclosed Integrator #1
Oct 10, 2016

that's awesome, as we have been sweeping this issue under the rug and silently working on it with our own system. if possible i would love to toss my name in the hat to have them test out some fixes with it.

JH
John Honovich
Oct 10, 2016
IPVM

Exacq has asked for info to be sent to support@exacq.com with subject: Case:173959 IPVM Hikvision time sync

They'd like to know:

  • Jikvision firmware version
  • Operating system on the exacqVision system
  • Serial number of the exacqVision system (or let us know its on a third party server)

Finally, they mentioned that the Exacq test record for that Hikvision camera is here.

U
Undisclosed #4
Oct 14, 2016
IPVMU Certified

"Finally, they mentioned that the Exacq test record for that Hikvision camera is here."

Unfortunately it says nothing about testing time sync.

UI
Undisclosed Integrator #2
Oct 11, 2016

Undisclosed #1... Let me know how you make out. I have already spent 6+ hours on this. 1 tech says something completely different than the other and I still have no fix. I have a support case with them already so I will continue with this one.

U
Undisclosed
Oct 11, 2016

I hope you're getting this resolved. In reading the thread I don't see anyone exactly (pun intended) ask this:

where is the primary NTP server (not some VMS camera time sync stuff, I mean a Real Network Time Server you can prove is present by using wireshark to observe the NTP traffic.)

Is the camera using the ntp server (not the vms, not "nothing"). P.s if the COMPUTER running the vms happens to also be the COMPUTER running NTP that'd be fine.

are you sure your NTP server is configured correctly. NTP servers get fussy if they don't have an outside time sync and time could start drifting.

are you sure all this video stuff (vms, onscreen camera display, tags in video, etc.) are using the NTP server time source for the time.

UI
Undisclosed Integrator #1
Oct 13, 2016

I have sent exacq the info as I was out of the office, but hopefully they will provide a resolution.

As far as your questions go, we have Axis cameras, and we have Hikvision cameras on the same system, the axis cameras are not missing a beat and are in sync with each other and the time server (which is local to the network, and we can communicate to it. The cameras are set to use the NVR, the firewalls are off (and have been for testing). The Hikvision cameras are off from eachother and the server by as much as 6 seconds, when you restart the exacq services and reconnect all cameras they are all synced up, but after a few days hikvision cameras are drifting. I have adjusted the time sync interval in the cameras to as low as 1 minute, and 90 minutes just to see if that would help (it didn't).

Hopefully exacq can figure it out or get us an answer.

UI
Undisclosed Integrator #1
Oct 13, 2016

I got a response back from exacq, they are working on it...

Thanks for the details. Our engineering team is investigating. I'll let you know when we have the answer. A couple of comments on your screen shots: I definitely see that Data Center and Main Entrance cameras are Hikvision and are off several seconds. On Tech Door, which is also Hikvision, the exacqVision time appears close to correct. If part of your concern is that the camera's timestamp is 4 hours off, that is by design. We always set the camera to UTC (EDT is UTC-4:00) and then use the server's timezone to compensate. Thus, that 4 hour offset is not indicative of an issue.

I responded that we set the hikvision time on the camera like #2 had done to prove the exacq time and camera time were different. And I also responded i knew about the 4 hour offset. Hopefully we will see what they come back with.

Avatar
Jeffrey Hinckley
Oct 14, 2016

I am a liitle curious as to why the time server is set to 127.0.0.1. This should be set to an accurate server, such as time.nist.gov. Check your camera to make sure it shows the actual ip address of the exacq server. Do not change any ntp settings at the camera. It must use GMT. I have seen that if you do not use an external ntp which the server can reach, like the default ntp.pool that exacq had programmed, cameras will be out of sync. I have also seen issues in the past where exacq will push a loopback address, 127.0.0.1, to the camera. This is from the network interface settings in windows, probably. If this is the case, usr the override feature with exacqvision server address. You would not be able to use cameras on a secondary nic. Make sure your dns settings and gateway are correct and working on server or this will not work. Never tried to run a NTP service on a server, and not sure why you would do this, since time will not be accurate on the system (and will drift over time).

If you have modified ntp settings on the camera, reset to default and start over. Leave camera ntp settings untouched. Make sure the server time settings are correct in windows (correct time zone) server and client.

UI
Undisclosed Integrator #1
Oct 14, 2016

I should clarify, i did catch that 127.0.0.1 as a time server, which actually got pushed to the cameras, and when i saw that address in the camera, I corrected it. The camera can connect to a time server, whether its pool.ntp.gov or the exacq box without issues, the problem is the cameras drift. We have defaulted cameras, the issue is not with a single camera, its with multiple cameras at different customer sites.

UI
Undisclosed Integrator #2
Oct 14, 2016

Just wanted to add my point. This is identically what is happening to me at multiple sites. I do have one site though that is close in the time sync. This has hik cameras that have old firmware.

Avatar
Jeffrey Hinckley
Oct 15, 2016

if you change ntp settings on the cameras, the exacq connection will, on a regular basis, over write it if ntp is enabled. Of course I am assuming that the camera NTP feature is working from a firmware standpoint. If you say the camera has the correct dns/ip information, and ntp functions, which is drifting, the time of the camera or that of the server? (you would have to deselect the camera in exacq so it does not keep pushing itself as the ntp server, unlesd the override has an alternate fqdn or ip address of another ntp source).

As I stated before, if you manually corrected the ntp on the camera, exacq will reset it on regular intervals (back to 127.0.0.1, or to itself, which will not work to keep accurate time). If you change settings such as frame rate, codec, and quality on the camera (those that I have used), these settings will be pushed back to Exacq. Can not be certain this is the case versions 7.0-8.0.

If the camera is showing 127.0.0.1 as the ntp server, though, I assume this is the problem since Exacq is pushing this out to the camera, which is trying to reference itself for the right time. Try entering the actual ip addresd of the server in the override field in exacq, and enable. Go to the camera to see if this is now listed as the ntp server. If so, turn on the osd time on the camera, and compare to the exacq server time, lower right corner of the client.

UI
Undisclosed Integrator #2
Oct 21, 2016

Hello Undisclosed # 1. Has Exacq come up with a solution yet?

UI
Undisclosed Integrator #1
Oct 21, 2016

Negative #2, I will reach out to them today.

UM
Undisclosed Manufacturer #5
Oct 21, 2016

So the reason that your time server is set to 127.0.0.1 is that you have anti-virus on the system. What happens is that the Exacq software asks the computer for what its IP address is so it can set the cameras NTP server to that value. The anti-virus software on the PC makes it so that instead of the real IP address it returns the loopback address. This is to prevent malicious viruses and malware from finding out what your IP address is and spreading. If you go to Configure System -> Date/Time there is a field ccalled Ip Camera Time Server: and an override checkbox. Click that box and then enter the NTP server, which I am assuming is your address.

(1)
UI
Undisclosed Integrator #1
Oct 21, 2016

#5, i have since corrected the 127.0.0.1, I set that in there by mistake in my initial testing i was trying to set the time server to itself, and should have used the actually IP of the server not the loopback address so I did catch that and corrected it only after I posted my issue here. I have used the time server override in my testing as well and on a live customer site to see if that resolved it. So disregard that i had 127.0.0.1 i have resolved that, restarted exacq services, cameras show proper NTP's and they can reach it, but we still have a time drift / sync issue i am waiting on exacq to troubleshoot. NVR is able to communicate, No AV software on servers, no firewall turned on, straight shot to internet. But again, this is an issue happening on all of my customers systems, not just isolated to one. These are straight out of the box exacq A series systems, or they are COTS servers running windows server or Windows 7.

GR
Greg Rhoades
Oct 21, 2016
IPVMU Certified

Have you tried getting the firmware from tech.hikvisionusa.com instead of the normal site?

UI
Undisclosed Integrator #1
Oct 21, 2016

I always get the firmware from that site, so yes I did get the firmware from that exact site.

UI
Undisclosed Integrator #1
Oct 31, 2016

So I finally heard back from exacq regarding the issue, and to my surprise there is actually an issue with the Hikvision side. I have not tested the fix, I am implementing it now on our in house system before we start sending it out the door.

TO ME-

The issue was traced to missing RTCP packets from Hikvision cameras. We have worked with Hikvision and they will be updating firmware to fix those versions where RTCP was not sending. In the exacqVision build linked below, the exacqVision server will detect when RTCP packets are missing and compensate by using the server time. Please let me know if this resolves your issue. This is planned for inclusion in our Dec 2016 release.

PS - chose the link appropriate for your platform - Windows 32 bit, Windows 64 bit, or Ubuntu.

Windows 32bit:

https://crm.exacq.com/release/Vision%20Server/8.1.x/exacqVisionServerDev-8_1_19_105016.exe

Windows 64 bit:

https://crm.exacq.com/release/Vision%20Server/8.1.x/exacqVisionServerDev-8_1_19_105016_x64.exe

Ubuntu:

https://crm.exacq.com/release/Vision%20Server/8.1.x/exacqVisionServerDev-8_1_19_105016.deb

Dan Rittman / Sr Director of Engineering / Tyco Security

So UD#2, if you want to give those links a shot, see if that helps you out, and i am going to do the same.

(2)
UM
Undisclosed Manufacturer #5
Oct 31, 2016

Wow, Dan Rittman personally sent that to you? He is one of the founders of Exacq. Must be a pretty serious issue.

UI
Undisclosed Integrator #1
Oct 31, 2016

He did indeed send that to me personally

UI
Undisclosed Integrator #2
Oct 31, 2016

Thanks so much UD 1. I have spent countless hours on this and I was run around the block and back and starting to loose faith. I hope this works.

UI
Undisclosed Integrator #1
Oct 31, 2016

I pretty much lost hope as well, we set a scheduled task to restart exacq services in the middle of the night which forces the cameras to reconnect and get pushed the time, which was a crappy work around but hid the issue. I just rolled the update out to our in house exacq box, and i will check it tomorrow to see how she works.

UI
Undisclosed Integrator #1
Nov 03, 2016

I just wanted to give a quick update, we installed the updated server on our in house box, and the cameras have held tight for a couple of days now, down to the second, so I am going to say problem soled. #2 how did you come out on your end?

UI
Undisclosed Integrator #2
Nov 25, 2016

I just installed on my first site. Only been installed for 1 hour so I will keep you posted.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions