Problem With Milestone VMS & Axis M3026 Camera?

We recently added 26 Axis M3026-VE cameras to a Milestone Enterprise system and the playback is choppy/not smooth. It looks like one of those old video games where the guy doesnt move smoothly and kind of skips through the screen. There is 16 existing IQeye cameras that are working perfectly, we installed a brand new server, cisco switches, and its on a dedicated camera network. We have tried pretty much everything (including suggestions from Milestone support) and have yet to come up with an answer. Here is some more information:

System Info:

  • Milestone Enterprise 2014 8.6c
  • Device Pack 7.8
  • Axis M3026 (5.55.1 and 5.55.13)
  • IQeye (Mix of different models)
  • Video Stream H.264
  • 7 FPS

Hardware:

  • Windows Server 2012 R2
  • Dual Xeon E5-2620
  • DDR3 1600 MHz 16 GB
  • Gig NIC
  • SSD for live archiving
  • WD Purple for archiving

Carlo, can you summarize Milestone's suggestions so that we don't waste your time recommending the same thing?

Here is what we tried so far:

  • New device pack installed
  • FPS was changed from 5 fps up to 15 fps
  • H.264 was changed to MJPEG
  • Constant bitrate was increased
  • Constant bitrate was changed to variable
  • Cameras were deleted and re-added
  • GOV was changed
  • Key Frames only option was removed
  • UDP to TCP changed
  • 50% of cameras were disabled
  • Network was checked
  • Server is not VM
  • No bottleneck on any system component
  • Cameras were separated between drives
  • No Firewall or Antivirus software running on the system
  • Wireshark and milestone diagnostics was submitted to Milestone
  • We have a physical Gig Lan, not a VLAN

Wow, that's pretty comprehensive.

Also, you've verified that the video is recorded at its full configured frame rates and without drops? And the CPU on the client machine is not maxing out when playing back the video?

This is definitely strange considering the IQinVision cameras don't have the same problem.

I am going to send this out to our discussion email list to get more suggestions.

yes, quite comprehensive. CPU is not maxing out when we are playing back.

Question is: does the camera really achieve to deliver the stream on the hard disk

On 3MP and 5MP sometimes , especially when you use 2 or 3 streams and Motion detect/ tamper, camera can saturate and don't supply the expected fps (despite the stats in camera)...

Can test by direct conenction and VLC fps measurements

second question : what 's about the recording ... and simultaneous playback ...

I notice you said SSD.... so you record live on SSD ? and try to read it in the same time.....?

Not sure SSD I/O buffer and performances are enough to supply wjat a regular Raid controler supplies

You can test a new recording path with same settings to a standard NAS and compare your performances. tell us what are Read/Write perf for your SSD

Marc,

We are using one stream per camera and the cameras are delivering everything up to the HDD properly.

SSD has enough I/O buffer, plus speed of SSD is fast so this buffer does not fill up (as can be seen through the disc queue). SSD speeds are averaging 550 (read and write).

We are exporting the video to check the play back.

FYI, we've been working with Milestone tech support for over 4 weeks now... 4 weeks.

Two options: when you record you are losing frames: camera are saturated (as you say that servers and netwwork and storage are ok)

Or .... somebody is slow somewhere

I 've been told that high capacity storage disk got problems whatever the brand is (> To disks) is that your case ?

Can you check if the camera's are delivering a continuous stream? We are currently having the same issues with Axis P1347-E and Q6045-E camera's, they are also delivering choppy video. When playing back frame by frame we noticed the gap between the frames is not equal. Eg. when using 10fps you would expect a frame every 100ms, but sometimes two frames are sent within 5ms and it takes 200ms for the next frame. The VMS we use is SeeTec. We tested in Milestone as well, but got the same result, so it seems to be camera related. This issue is more noticeable as the amount of movement increases or the scene is more complex (eg. city surveillance in a busy street)

For now nor SeeTec or Axis have been able to solve this.

Switch issue

do not see any switch models numbers, switch throughput would be my best guess. Say no bottleneck, has a snifer analized the network?

I agree with 'A' in that some Axis cams have issues when there is motion in the view. I can corroborate 'A' observation with a P1347.

To go along with what MikeR said... do you have one of the single POE injectors or even another small POE switch that you can use to run just the ONE camera direct? This would be from the Server to the POE to the cam... that's all.

The fact that you are using the Server to view with is typically not recommended as it exposes the system to dropped video going to storage because the CPU is too busy handling the viewing function. Be mindful of the salvo sizes when you set them up and monitor the CPU and memory in order to strike a balance. Be sure to watch the HDD queue lengths and keep them small. If they are less than one...great. RAID volumes allow for larger values temporarily.

Also, Axis cams can be told to output internal info in the stream which lets you observe what it is trying to do. I like to use the #b and #r ones. A doc about this is here. Axis Text Overlays

Axis video display isn't the reality, just what the camera thinks she is sending ...

you can see it when setting Mjpeg , 30 fps , 0% compression , you can reach 130 Mbits with %B but when measuring the real throughput ...with bandwidth utility or with VLC on the stream it's an other story . What it is saying isn't what you can get, especially when camera technically collapses. So better use VLC on the stream you wanna test. VLC measure bandwidth but also real frame rate. Motion detect adds also camera CPU saturation issues...

I agree...that's why I said 'what it is trying to do' in my post. The result at whatever destination software is what counts.

Apparantly Milestone came up with a fix for our issue. We implemented it on one of the cameras last night and it looks like it resolved the issue. He is going to apply it across all the cameras and we will see if we get the same success. I will post the result once I get an understanding of what was done.

Please do. This is a nightmare scenario for installers. Glad you got an answer. I have followed the issue closely.

Carlo, is that a software upgrade / patch from Milestone or?

I am not sure what they sent to our tech support team, but my understanding is that it is not a patch or upgrade. I undertstood it to be settings inside the Milestone software. Milestone replicated it in their environment (same software and camera model) and found the same results.

I will post a full update once it gets resolved with step-by-step instructions so that others can benefit.

I would suspect a switch or switches. I saw similiar issue in a large facility with only a few cameras. After pulling out most of the hair I have left, I discovered that they had added a VOIP system and we were sharing the same switches. We replaced those switches, including the core switch, with better/newer equipment and the problem went away. I don't know that it was a single switch or degradation across multiple switches. In a large facility it is hard to tell.....

I had a similar issue using a different VMS with that same camera. Using H.262 in the camera switched H.264 from main to baseline and the problem went away.

I have had the same issue with a few Axis cameras with "choppy" video / unequal frame spacing. The P3367 (5MP dome) will give consistently choppy video if used in 5MP mode and pointed at a complex scene. Simply taking the camera and pointing it against a white wall solves the problem. As an example, I have one of these pointed at an area where about half the view is a crushed rock parking area (you'd be surprised how much more bandwidth this uses than asphalt) and the other half is trees that are constantly swaying in the breeze. A person walking through the frame is never smooth.

I spent weeks troubleshooting this years ago (units were RMAd, new firmwares, connect directly to camera with no networking equipment, change GOP, etc.) with no success.

On the plus side, the image quality is great and low light performance is acceptable - particularly for a 5MP camera that's now going on 4 yrs old.

If Milestone was able to solve this without changing to MJPEG or reducing the resolution, I would love to hear about it.

Thanks.

Can you try to view playback in milestone 2013 version as i have face simillar problem. I kept getting jerks on playback. At the end it was found that Milestone 2014 smart client has some issue. Kindly try to view playback on Milestone 2013 smart client.

Well what I thought was a resolution turns out to be somewhat of a work around (not really a good one either). We were told by Milestone support:

- Go to camera property, control priority and beside bitrate choose Frame Rate as the priority over quality

Although this fixed the issue, it really degraded the video quality. so we are kind of in the same boat, different problem.

I will have to play with this in my lab. What happens if you now change the camera side compression down to a smaller number? I recall Axis like to start at a 30% value and you can take it down to 1%.