Subscriber Discussion

Problem With Upgrading Milestone 2014 To 2016 - PTZ Jitter And Delay

UE
Undisclosed End User #1
Oct 15, 2017

Hello

I recently upgraded Milestone Xprotect Corporate 2014 to 2016R3. I have AXIS PTZ and Avigilon Fixed cameras. Soon after the upgrade I noticed that there are jitters and delay in playback. I noted following things:

1) Playback is normal but when I playback the video there is jitters and delay in video.

2) Recorded video is ok because when I export the video, exported file is normal as it should be.So problem is with realtime playback.

3) I face this issue on random cameras on different recording servers. Also note that on a single recording server some cameras are normal and some cameras I face this issue.

4)The playback issue is on PTZ cameras only, not on fixed cameras.

5) When I playback the video stored before the update , the video is normal, but as it shifts to timeline after the update, There are jitters and delay in playback.

Did anyone else faced this issue ?

UE
Undisclosed End User #1
Oct 15, 2017

I am using AXIS Q6045E MKII camera.

U
Undisclosed #2
Oct 15, 2017
IPVMU Certified

Is that the only model with jitter?  How many do you have? Do all of them do it all the time?

UE
Undisclosed End User #1
Oct 15, 2017

We are using axis Q6045E which is having this issue. But there is not jitter with live streaming. We face jitters when we pull streams to view in playback. We have total 1000 cameras but most of them are giving this issue.

U
Undisclosed #2
Oct 15, 2017
IPVMU Certified

Is the jitter only when using the console for playback, or web client/mobile?

UE
Undisclosed End User #1
Oct 15, 2017

I am facing jitters when I playback through Smart client (2014R3,2016R3 and 2017R2). I dont have mobile app as its against our organization policy.

U
Undisclosed #2
Oct 15, 2017
IPVMU Certified

Have you applied this hotfix:

UE
Undisclosed End User #1
Oct 16, 2017

I installed 10.2b inwhich this hotfix was included. Still I already tried this hotfix but unfortunately it could not solve my problem.

U
Undisclosed #2
Oct 16, 2017
IPVMU Certified

Just to see if it has any effect, have you disabled hardware acceleration or increased the size of the video buffers?

Btw, It's likely that Joshua Hendricks from Milestone will see this tommorow...

Avatar
Josh Hendricks
Oct 16, 2017
Milestone Systems

Since you're having this issue only on one specific model of camera, I don't think it will be related to the hotfix rollup previously mentioned. Though, if you're not running version 2016 R3 SP1 (10.2b), it is definitely important to apply that hotfix.

I could be wrong, but it sounds like perhaps there was a change in behavior in the way the Recording Server delivers recorded video and perhaps an underlying issue with the video stream from these cameras is surfacing as a result.

I have seen this type of behavior for example when a camera is sending "jittery" time stamps with the video stream; oddly variable time deltas, two consecutive identical time stamps, or even a time stamp older than the previously received time stamp. Live is generally unaffected because we try to push the image out to clients as soon as we receive it. But playback will be more sensitive to this behavior.

I'm not saying it's a problem with the camera, only that it would be the first thing we should investigate based on the symptoms provided (live is fine, exports are fine, playback for other cameras is fine, only playback for these cameras are affected).

If you haven't already opened a case with us on this, please feel free to copy my hypothesis and attach it to your case. Some things we'll want to collect to help us diagnose the issue may include...

  • Milestone Diagnostics Tool and Milestone Analyzer results from the Recording Server
  • Wireshark trace between Recording Server and one of the affected cameras for maybe 15 seconds
  • A zipped copy of an XProtect format export of an affected camera (maybe 5 minutes tops)
  • If possible, a copy of an actual archive table from an affected camera. This is ideal as we can then have R&D investigate the block files frame by frame and determine whether there are "invalid" time stamps that could be causing this behavior. These folders can tend to get pretty big though, so see if you can work with our support team to get a folder zipped up and sent over for analysis. Alternatively depending on your availability/accessibility we can arrange for R&D to review the system remotely

Some things you probably know we're going to ask include...

  • Are you using the most recent Device Pack version?
  • Is the firmware for the affected cameras the most recent tested firmware version?
  • If a newer firmware version than we have tested is available, it may be beneficial to see if there is an improvement when upgrading to the most recent firmware version, even if it has not yet been tested by Milestone (Axis cameras usually roll back firmware gracefully when needed).

Looking forward to seeing what the ultimate cause was. I don't expect you'll have any issues but feel free to reach out to me at jh@milestonesys.com if you have any difficulties.

(3)
Avatar
Josh Hendricks
Oct 16, 2017
Milestone Systems

I just noticed your account type in IPVM is identified as an end-user account? If you have Care Premium, definitely give us a call and provide your Milestone Account ID. Otherwise please contact your reseller and provide them the information from this thread if you have not done so already. They will bring us in to have a look at the issue. If you have any questions, feel free to ask them here or to email or call me directly.

Josh Hendricks
jh@milestonesys.com
+1 503 350 1174

(1)
U
Undisclosed #2
Oct 16, 2017
IPVMU Certified

Joshua, does Milestone assign timestamps based upon when it receives the frame (server timestamp) or based on the camera sends the frame (camera timestamp)?

Also, no matter how off the timestamps may be, wouldn't you expect that an export would contain the same jitter/judder as normal playback? And yet, he says exports are fine...

Avatar
Josh Hendricks
Oct 16, 2017
Milestone Systems

Also, no matter how off the time stamps may be, wouldn't you expect that an export would contain the same jitter/judder as normal playback? And yet, he says exports are fine...

Yes and no. When you playback in Smart Client from a live system, the Recording Server is sending you the frames to be rendered. When you playback from an export, the Smart Client Player is reading the data directly from disk. So it's possible that there could be a minor difference in the way the data is read/rendered depending on the source. That's what we'd want to get to the bottom of in this case.

I don't have enough concrete knowledge about the internals of media storage to answer the first question, but our team can certainly help get to the bottom of it.

(1)
UE
Undisclosed End User #1
Oct 22, 2017

Dear Joshua

Thank you so much for your response and help

The case is already opened with milestone(MSC260420).

We are having 2 types of cameras

A- AXIS PTZ camera

B- Avigilon fix camera

on axis ptz cameras sometimes we get less than 1 fps

during play back of fix cameras we get frame loss of 4 to 5 fps.

So PTZ camera's behavior is worst.

 

All Milestone higher management is involved in the case now.

UE
Undisclosed End User #1
Oct 22, 2017

I checked by disabling the hardware acceleration. The behavior is weird as sometimes the same footage plays normal but next time i get less than 1fps for same footage. If you open my case then may be you are already aware of the problem.

 

Thanks

UE
Undisclosed End User #1
Oct 22, 2017

FYI,

We upgraded 2016 to 2017R3 before one week in presence of Milestone support team on site. Our propose for this quick update was to solve the problem but we were unlucky that same problem we faced in the updated version. All our firmware and device pack are updated and tested.

UE
Undisclosed End User #1
Oct 24, 2017

Joshua ?

Avatar
Josh Hendricks
Oct 24, 2017
Milestone Systems

Some of our most experienced engineers are working with you on the case at present. It certainly could be a software issue, but I know there were a number of concerns raised relating to the network and NIC configuration with some actions on your end which are currently pending. 

Of course if an issue affects only a subset of cameras, it can seem like the network is an unlikely culprit, and that may be true, but the network is fundamental to the success of any VMS so it is important that we have thoroughly checked everything - especially in a difficult case like this where the basic steps that solve 90% of similar issues have not been successful.

My advice is to continue to cooperate with our team in EMEA who are heavily invested in your success. Trying to assist you in parallel from these forums would be counter-productive. Hopefully we will soon find a solution and help to put this issue to rest.

I do wish you all the best, and I certainly understand that you're reaching out here after a long stretch of troubleshooting with our support team. I will check with my colleagues overseas to see if there's any value I can add to the case on the back end but they are a talented team over there and it looks like the right people are involved already.

U
Undisclosed #2
Oct 24, 2017
IPVMU Certified

Joshua, was there ever a resolution to this similar sounding problem?

Avatar
Josh Hendricks
Oct 24, 2017
Milestone Systems

It looks like that user solved the issue by disabling Transcoding in their views and/or disabling hardware acceleration. More than likely it was just disabling transcoding (changing quality to Full) that actually solved it, but the OP never commented back it appears.

Transcoding should be used sparingly and with intent. The reason it wasn't enough in his case to disable it in the Smart Client Profile is because the profile can be used to limit which features are available in the Smart Client, which can prevent folks from using them. Changing the features available for use does not change already configured items including views. Being a developers forum there is/was probably a programmatic method available to modify all existing views to end the use of transcoding.

As much as I'd like this customer's issue to have such a quick and easy solution, this is something that is typically discovered on the first call and/or use of our analysis tools, and explicitly disabling hardware acceleration is another typical troubleshooting.

U
Undisclosed #2
Oct 24, 2017
IPVMU Certified

...but the network is fundamental to the success of any VMS so it is important that we have thoroughly checked everything - especially in a difficult case like this where the basic steps that solve 90% of similar issues have not been successful.

Yes, but in this particular case couldn’t one simply test for glitchy playback by running Smart Client on the Server itself?

Possibly with the network cable pulled out?

If it still glitches its probably not the network, right?

Avatar
Josh Hendricks
Oct 24, 2017
Milestone Systems

Yes, but in this particular case couldn’t one simply test for glitchy playback by running Smart Client on the Server itself?

Possibly, unless the issue comes down to jitter between these particular cameras and the Recording Server. If the live view is rendered using the h264 timecodes in the stream from the camera, but playback is rendered by the time of frame arrival (I do not have the answer to this right now), then I could see the issue still happening even with the client running directly on the Recording Server. You are never truly independent from the network unless maybe your video source is a webcam or PCI based encoder card.

U
Undisclosed #2
Oct 24, 2017
IPVMU Certified

If the live view is rendered using the h264 timecodes in the stream from the camera, but playback is rendered by the time of frame arrival (I do not have the answer to this right now), then I could see the issue still happening even with the client running directly on the Recording Server.

Fair enough, but remember exported video is smooth; would export  use order of arrival, like live?

In any event, U1 should be able to go frame by frame, (using the right/left arrow keys if I remember correctly), and look at the timestamp shown for each frame, using local Smart Client.  If they are consistently spaced as one would expect for the cameras frame rate, but hitting play renders choppy, then presumably it’s not network related, yes/no?

Were there any Media Overflow errors in the logs that you are aware of?

Avatar
Josh Hendricks
Oct 24, 2017
Milestone Systems

I'm not sure if exports will use the exact same methodology as live or playback to temporally space out the frames. But I know not all cameras or streaming protocols provide timecodes in the stream, so I'm fairly certain it would vary by device driver whether we use the timecodes from the camera or whether we use arrival time for frame storage. It's possible that exports could use the timecodes from the database files in the export while the Recording Server might use arrival time or vice versa. Hopefully they would be using the same strategy, and they probably are, but I just mention it as a possible explanation for the difference between exports and playback.

 

With regard to media overflow, I don't know if EU1 has an issue with Media Overflow but it would not cause the described symptoms. If there is media overflow, it means there are frames being dropped because the data cannot be written to storage as fast as it arrives. Each channel of data (camera/mic/metadata) has a memory buffer/queue for incoming data to accommodate for brief increases in incoming data or brief storage latency, and if this buffer reaches it's size limit (configurable), frames will be discarded until the queue is no longer full. The effect would be that live would be smooth, but both playback and exports would be choppy due to missing frames.

In this case, if I'm not mistaken, playback is closer to keyframes only while exports are full frame. There are too many factors to consider, and facts about the case and the environment that I'm not privy to at the moment to comment on whether it could or could not be network related. All I know is that there have been network related issues in the past, and there are currently some concerns that are being addressed. I'll leave the solving of this case to my colleagues "on the ground", but I'm sure the eventual solution will be of interest to the community and I see value in it being shared here if EU1 sees fit to do so!

 

U
Undisclosed #2
Oct 24, 2017
IPVMU Certified

Thanks Joshua, good points, for sure.

One final question for U1 and your comment as well:

Is the choppiness exactly repeatable?  Meaning when you replay an exact segment of a clip does it glitch in exactly the same places for the same time?

If it doesn’t, that would imply something dynamic is going astray, as opposed to network/timestamps, Yes/no?

Avatar
Josh Hendricks
Oct 24, 2017
Milestone Systems

that would imply something dynamic is going astray, as opposed to network/timestamps, Yes/no?

Hmmm, maybe? :)

If it's glitching the same every single time, then it could be related to the network between camera and recording server, or the content of the video stream itself. The predictability of it certainly helps to narrow the scope of the issue.

If it's glitching differently every time, then it might be possible to rule out everything between the camera and recording server.

You have some good instincts for troubleshooting; we have some open positions for support/solution engineers in the US and abroad ;)

U
Undisclosed #2
Oct 25, 2017
IPVMU Certified

You have some good instincts for troubleshooting; we have some open positions for support/solution engineers in the US and abroad ;)

That’s awfully kind of you, but I work for votes.  Well, usually ;)

(1)
UE
Undisclosed End User #1
Oct 26, 2017

No, as i told before the problem is not consistent.

For example.

1)I select a time period of playback , there is jitters and choppiness in video

I will select one day or two days back and then comes again on that time period , the video is smooth now.

2) If there is choppiness in playback, sometimes If i move the time line slider little bit forward of backward, the video playback becomes.

3) Sometime if there is choppiness, If I pause the video and play again after some time , the video become smooth for sometime.

 

4) If there is choppiness , sometimes if i simply pause it and go frame by frame using next frame button, after pressing 3,4 times the same button and playagain, the video becomes smooth.

5) Sometimes if I leave the choppy video to play for 7,8 minutes, it becomes stable, but if I change another time again I face same issue.

6)sometimes non of above work around works for me.

 

U
Undisclosed #2
Oct 28, 2017
IPVMU Certified

EU1, something that I’m sure you have considered, but I feel compelled to mention, since it could account for most of the behavior above, is a fault in the SAN or storage.  The playback might improve or degrade based on caching, as you repeat the segment.

FWIW, this is what I would do:

Create from scratch on a new machine with local disks the smallest, simplest configuration, with 1 or 2 cameras on an isolated switch, using the exact version of Xprotect you have.

If it has the same problem, tell Milestone to replicate it in their lab.  Once they can replicate it, I would think they could figure it out very quickly.

If you don’t have the problem there add things back in until you do have a problem pointing to the culprit.

 

 

UE
Undisclosed End User #1
Oct 26, 2017

We are using the network infrastructure designed by Milestone, eventually the best equipment with 10G links. Till now we have not found anything during hard troubleshooting for last one and half month.

There is no frame drop from camera to storage,

But recently I noticed one counter value that is exceeding the limit on any camera that has problem during the playback. I already informed the higher technical management of milestone and waiting for their reply.

UE
Undisclosed End User #1
Oct 26, 2017

Yes, During playback if I go next frame by frame, I can see all the frames. But issue is in smooth playback.

UE
Undisclosed End User #1
Oct 26, 2017

I installed the client directly on recording server and faced the same issue... The issue is still not solved. The higher management from milestone is still working.

The weird thing is the issue is not consistent, not specific to one camera only. The cameras are changing randomly. Video is taking time to load.

Avatar
Mike Dotson
Oct 25, 2017
Formerly of Seneca • IPVMU Certified

A couple other things to pay attention to:

1) Where is the playback data coming from...the LiveDB or the ArchiveDB.   These are usually arrays that have different performance metrics.

2) Does your Client machine support Intel Quicksync?  

If not, try the steps in this KB...Article number:  000002403 to decode streams to old way.

GPUZ is a good tool to watch a systems GPU resources.

UE
Undisclosed End User #1
Oct 26, 2017

Dear Mike,

1) the issue is with live db also which is attached directly with RS through SAN network. that is why we have not found anything on network side. Video from camera to RS and then to live db is ok as exported file is smooth.

2) we are running client application on many systems. All of them using intel core i7 6700. But hardware acceleration is off in smart client.

 

UE
Undisclosed End User #1
Oct 26, 2017

One thing I have noticed during the troubleshooting that when there is choppy video then "Render queue underflow count" starts increasing and sometimes increases more than 60,000.00 to 80,000.

I compared this value with other cameras which were not having problem with playback then this value was even less than 100 during the playback.

Just a question from the experts that what is this value and could it be the reason ?

 

Thanks

 

Avatar
Mike Dotson
Oct 26, 2017
Formerly of Seneca • IPVMU Certified

Another thing that should be looked at is if the server is running the Archiving function at the time of the choppiness.   This is the fcn that collects the recent LiveDB data and moves it to the ArchiveDB.

It will do massive Reads from the LiveDB and massive Writes to the ArchiveDB.   This will clog up access during this time quite easily.

In the Management Client...the Current Task choice (by System Monitor) will let you see if this is happening.

Another slight possibility is the task that processes the 'expiration' of data or the 'auto-archive cleanup'.     You have to look in the log files out on ProgramData to see this activity.   The specific logfile name will have 'Database' in it and the messages will have 'Time Limit' or 'Size limit'.

UE
Undisclosed End User #1
Oct 28, 2017

We have already checked that possibility. It was not archiving at the time.

Avatar
Mike Dotson
Oct 26, 2017
Formerly of Seneca • IPVMU Certified

 Milestone also has a doc that describes how to activate a diagnostic on the SmartClient that shows how the data is being handled.

This is a link to that doc:  SmartClient-diag

The level-2 described shows the basic detail on GPU.  Level 3 shows what is coming in from the server as well as the rendering engine.

UE
Undisclosed End User #1
Oct 28, 2017

Mike in level 3, what is Render queue underflow count value ? I have seen that this counter goes more than 100,000 when there is choppiness in video while its less than even 100 when video plays smooth.

UE
Undisclosed End User #1
Oct 29, 2017

@Joshua,

Sir can you please explain what is render queue overflow count and render queue underflow count ?

 

Avatar
Josh Hendricks
Nov 02, 2017
Milestone Systems

Hi E1, I'm glad the issue seems to have been discovered and resolved! To answer your question, here is a brief explanation of these values:

Render queue overflow = Total number of frames that have been dropped after decoding because they expired before being requested.

Render queue underflow = The amount of requested video frames which were not already cached when asked for.

Hopefully this gives you a general idea of the meaning. If you require more technical detail about these, I'm afraid I'll have to team up with someone in R&D with more knowledge about the rendering process.

U
Undisclosed #2
Nov 02, 2017
IPVMU Certified

Joshua, can you confirm the discovery of this bug and hotfix on your end?  

It sounds like it could affect playback of any camera using edge storage and the new version.

UE
Undisclosed End User #1
Nov 05, 2017

I will be very glad if you will provide me the detail of these two counters.

And also if you can explain deeply what was the actual bug.

Avatar
Mike Dotson
Oct 30, 2017
Formerly of Seneca • IPVMU Certified

Now you are getting into questions that also need to have the hardware you are running on known.

For a good hardware acceleration experience (in general) on the client, you need

1) good NICs ( we prefer Intel)

2) A video card to support the rendering out to the monitors.  Only connect the monitor to the video card, enable multi-monitor in the BIOS, Be running Win8.1 or higher.

3) The memory needs at least one set of dimms to cover all CPU memory channels

4) CPU...you have already mentioned the i7-6700...good one.

There is another KB article that is for troubleshooting client performance that may assist..Article number:  000003359.

 

(1)
UE
Undisclosed End User #1
Nov 02, 2017

Mike,

Our client computers are already higher in specification.

The good news is, the problem has been solved today morning. There was a bug when they upgraded from 2014 to 2016. The development team of Milestone trapped the bug after collecting a series of logs and then release hotfix.

We had edge recording on all of our devices. Milestone had a bug, as I understood, that it was continuously retrieving edge recording and maintaining a separate parallel video database. When we used to playback the videos, the recording server was mixing both database and in result there was extra pipelines usage and frame drop.

The reason why we were facing the issue on PTZ cameras was, we only enabled edge recording on PTZ.

We were also confused that why some of the PTZ cameras on same recording server are working and some , of same type  and same settings, are not working. The reason was also that edge recording was disabled.

 

Now Alhamdullilah the problem is solved with the hot fix.

 

 

Thanks to all of you.

 

(1)
U
Undisclosed #2
Nov 02, 2017
IPVMU Certified

We had edge recording on all of our devices. Milestone had a bug, as I understood, that it was continuously retrieving edge recording and maintaining a separate parallel video database. When we used to playback the videos, the recording server was mixing both database and in result there was extra pipelines usage and frame drop.

Glad you got it fixed!

So if you had attempted playback when the camera was offline, would that have been smooth?

UE
Undisclosed End User #1
Nov 05, 2017

No

We didnt attempt to play back when the camera was offline.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions