Carlo, can you summarize Milestone's suggestions so that we don't waste your time recommending the same thing?
Here is what we tried so far:
- New device pack installed
- FPS was changed from 5 fps up to 15 fps
- H.264 was changed to MJPEG
- Constant bitrate was increased
- Constant bitrate was changed to variable
- Cameras were deleted and re-added
- GOV was changed
- Key Frames only option was removed
- UDP to TCP changed
- 50% of cameras were disabled
- Network was checked
- Server is not VM
- No bottleneck on any system component
- Cameras were separated between drives
- No Firewall or Antivirus software running on the system
- Wireshark and milestone diagnostics was submitted to Milestone
- We have a physical Gig Lan, not a VLAN
yes, quite comprehensive. CPU is not maxing out when we are playing back.
Question is: does the camera really achieve to deliver the stream on the hard disk
On 3MP and 5MP sometimes , especially when you use 2 or 3 streams and Motion detect/ tamper, camera can saturate and don't supply the expected fps (despite the stats in camera)...
Can test by direct conenction and VLC fps measurements
second question : what 's about the recording ... and simultaneous playback ...
I notice you said SSD.... so you record live on SSD ? and try to read it in the same time.....?
Not sure SSD I/O buffer and performances are enough to supply wjat a regular Raid controler supplies
You can test a new recording path with same settings to a standard NAS and compare your performances. tell us what are Read/Write perf for your SSD
We are using one stream per camera and the cameras are delivering everything up to the HDD properly.
SSD has enough I/O buffer, plus speed of SSD is fast so this buffer does not fill up (as can be seen through the disc queue). SSD speeds are averaging 550 (read and write).
We are exporting the video to check the play back.
FYI, we've been working with Milestone tech support for over 4 weeks now... 4 weeks.
Two options: when you record you are losing frames: camera are saturated (as you say that servers and netwwork and storage are ok)
Or .... somebody is slow somewhere
I 've been told that high capacity storage disk got problems whatever the brand is (> To disks) is that your case ?
Can you check if the camera's are delivering a continuous stream? We are currently having the same issues with Axis P1347-E and Q6045-E camera's, they are also delivering choppy video. When playing back frame by frame we noticed the gap between the frames is not equal. Eg. when using 10fps you would expect a frame every 100ms, but sometimes two frames are sent within 5ms and it takes 200ms for the next frame. The VMS we use is SeeTec. We tested in Milestone as well, but got the same result, so it seems to be camera related. This issue is more noticeable as the amount of movement increases or the scene is more complex (eg. city surveillance in a busy street)
For now nor SeeTec or Axis have been able to solve this.
do not see any switch models numbers, switch throughput would be my best guess. Say no bottleneck, has a snifer analized the network?
Seneca | IPVMU Certified | 03/04/15 10:34pm
I agree with 'A' in that some Axis cams have issues when there is motion in the view. I can corroborate 'A' observation with a P1347.
To go along with what MikeR said... do you have one of the single POE injectors or even another small POE switch that you can use to run just the ONE camera direct? This would be from the Server to the POE to the cam... that's all.
The fact that you are using the Server to view with is typically not recommended as it exposes the system to dropped video going to storage because the CPU is too busy handling the viewing function. Be mindful of the salvo sizes when you set them up and monitor the CPU and memory in order to strike a balance. Be sure to watch the HDD queue lengths and keep them small. If they are less than one...great. RAID volumes allow for larger values temporarily.
Also, Axis cams can be told to output internal info in the stream which lets you observe what it is trying to do. I like to use the #b and #r ones. A doc about this is here. Axis Text Overlays
Apparantly Milestone came up with a fix for our issue. We implemented it on one of the cameras last night and it looks like it resolved the issue. He is going to apply it across all the cameras and we will see if we get the same success. I will post the result once I get an understanding of what was done.
I am not sure what they sent to our tech support team, but my understanding is that it is not a patch or upgrade. I undertstood it to be settings inside the Milestone software. Milestone replicated it in their environment (same software and camera model) and found the same results.
I will post a full update once it gets resolved with step-by-step instructions so that others can benefit.
I would suspect a switch or switches. I saw similiar issue in a large facility with only a few cameras. After pulling out most of the hair I have left, I discovered that they had added a VOIP system and we were sharing the same switches. We replaced those switches, including the core switch, with better/newer equipment and the problem went away. I don't know that it was a single switch or degradation across multiple switches. In a large facility it is hard to tell.....
I had a similar issue using a different VMS with that same camera. Using H.262 in the camera switched H.264 from main to baseline and the problem went away.
I have had the same issue with a few Axis cameras with "choppy" video / unequal frame spacing. The P3367 (5MP dome) will give consistently choppy video if used in 5MP mode and pointed at a complex scene. Simply taking the camera and pointing it against a white wall solves the problem. As an example, I have one of these pointed at an area where about half the view is a crushed rock parking area (you'd be surprised how much more bandwidth this uses than asphalt) and the other half is trees that are constantly swaying in the breeze. A person walking through the frame is never smooth.
I spent weeks troubleshooting this years ago (units were RMAd, new firmwares, connect directly to camera with no networking equipment, change GOP, etc.) with no success.
On the plus side, the image quality is great and low light performance is acceptable - particularly for a 5MP camera that's now going on 4 yrs old.
If Milestone was able to solve this without changing to MJPEG or reducing the resolution, I would love to hear about it.
Can you try to view playback in milestone 2013 version as i have face simillar problem. I kept getting jerks on playback. At the end it was found that Milestone 2014 smart client has some issue. Kindly try to view playback on Milestone 2013 smart client.
Well what I thought was a resolution turns out to be somewhat of a work around (not really a good one either). We were told by Milestone support:
- Go to camera property, control priority and beside bitrate choose Frame Rate as the priority over quality
Although this fixed the issue, it really degraded the video quality. so we are kind of in the same boat, different problem.