Subscriber Discussion

Do People Find Milestone Always Blames Hardware And Not Their Software?

UI
Undisclosed Integrator #1
Oct 20, 2016

I am starting to find every time I have a problem with a Milestone system they always start by saying it must be hardware!

(1)
(1)
JH
John Honovich
Oct 20, 2016
IPVM

That's why you need to buy Husky!!!! :)

In general, we hear quite a lot of software only providers cite hardware issues.

What problems specifically is Milestone saying must be hardware?

(1)
(1)
(1)
UI
Undisclosed Integrator #1
Oct 20, 2016

I have installed a Pro Milestone System running 2016 R2. I have the LPR add on running on this too. Total numbers of cameras is 63 running on 2 servers (Intel Xeon's)

I am getting a problem with the LPR side of Milestone. It’s logging the number plate on the database but if you click on one of the number plates in smart client its shows a different video clip. This does not happen every time. A few mornings ago between 0730hrs and 0803hrs it logged all the number plates coming through the gate but no video footage was recorded.

Milestone have said this is a camera problem! I said its logging the number plate so the camera is working ok. Then they said it’s because the camera is too dark.... It’s a LPR camera (Bosch Dinion 5000 IP) so it should be dark!!

Now they have said it’s the CPU on both servers... When they checked today both server ran about 70% CPU. When I check its running at 15% on both servers.... I logged this for an hour and it did not top 20%.

I am pulling my hair out. I have had this problem now for 3 weeks...

(1)
(1)
GR
Greg Rhoades
Oct 20, 2016
IPVMU Certified

We've started making our servers and workstations beyond their recommendations, both for expandability and exactly for this reason. If they even think about trying to point to that, I just pull up device manager or the system specs and performance monitor. That being said-Windows(like all software) is far from flawless.

As for your problem, have you made sure the time in the camera's settings is set to sync with the server?

UI
Undisclosed Integrator #1
Oct 20, 2016

I always over spec now because of this. Before contacting Milestone I made sure all cameras and servers looked at the same NTP server and also check the time is the same.

(1)
Avatar
Josh Hendricks
Oct 20, 2016
Milestone Systems

I work for Milestone

That's a tough one! If our staff is saying it's a camera problem I wonder if it is a misunderstanding of the reported issue? Reading it here, it doesn't sound like a camera issue at all. From the sounds of it, I would venture to guess it could be...

- dual streaming - if you use MJPEG for the live stream (ideal for analytics) but h264 for recording (ideal for storage), there can and often will be a discrepancy between the time we receive the same frame from the camera between the two codecs. Typically the h264 stream will arrive from .5-2 seconds AFTER the MJPEG image. So when you view the event, it's possible the box will be drawn "ahead" of the plate. If you're dual streaming, try single streaming temporarily to see if that's it. If that's the issue we can discuss possible methods improve the experience. Options might vary by camera as we don't always get a time reference in the video stream if I'm not mistaken.

- high CPU can delay the analytics and result in some queuing of frames to be analysed, though I would think the time stamp registered on the reading would be based on when the image was received (or generated) rather than when it was analysed.

For the unrecorded footage, are you recording full time or on motion/event? If you're not doing full time recording you might consider using a LPR list to trigger a manual event, and setup the software to record when the event is triggered, with a prebuffer. That way there should always be recordings if a plate was read.

(4)
UI
Undisclosed Integrator #1
Oct 20, 2016

Hi Joshua,

Thanks for your reply. It is my understanding that Milestone have checked this and also changed some settings. Would it be possible for you to look at the case? I it is open on your system.

Avatar
Josh Hendricks
Oct 20, 2016
Milestone Systems

I'll gladly make sure it gets another pair of eyes on it. I could not find a recent case matching this description but if you let me know the case number I will follow it up for you.

UI
Undisclosed Integrator #1
Oct 20, 2016

MSC213677

Avatar
Josh Hendricks
Oct 20, 2016
Milestone Systems

Thanks, I have asked our EMEA team to review the case. It's about 3:30am here in Melbourne or I'd offer to take a looksee. If unattended TeamViewer is an option, I'll gladly pop in later today if you can drop the remote access info in the case comments. Otherwise I'll see if I can add any value to the case notes later on.

UI
Undisclosed Integrator #1
Oct 20, 2016

Thanks for your help. All the Teamviewer login information is on the case notes. User name, password etc... You can login anytime.

Avatar
Josh Hendricks
Oct 20, 2016
Milestone Systems

The case has been updated, but for the benefit of future readers here with LPR issues...

I've done a brief review and here're my observations:

- CPU utilisation appears to be well within "normal", at least during my remote session
- Motion detection settings were at defaults
- Pre/post recording buffer were at 2 seconds each
- The plates are often only in the image for 1 second, maybe up to 3 seconds
- Recording is based only on motion detection

Here are the changes I made:

- Enabled manual motion detection sensitivity and cranked up the sensitivity
- Unchecked "keyframes only" in motion detection
- Created a global manual event on the slave server called "Record LPR Event" with a follow-up event called "Stop Recording LPR Event which fires 2 seconds later
- Configured the camera to record both on motion, and also on the above start/stop event triggers
- Added the "Network Service" user to the Administrators group in Windows on the Slave Server so that the Event Server can properly authenticate with the slave server and activate events remotely (Event Server is running as Network Service)
- Configured each of the License Plate Match Lists on the Master Server to trigger the "Record LPR Event" event which was setup on the slave server - that way any time a plate is detected, that event will be triggered and it SHOULD be recording for 2 seconds with a 2 second pre/post buffer (6 seconds total per plate)

And here's the commentary:

Due to the cars passing fairly quickly through the image, it is likely that motion detection was missing them all together. An image will only be recorded based on your current settings if we detect motion. And motion detection on H264 will be based on keyframes only by default (to reduce CPU utilisation). So it is possible that a plate was detected without also generating enough motion to trigger recording.

I enabled the manual motion detection options and turned the sensitivity up quite a bit. I also unchecked "keyframes only" for this camera, and set the motion detection interval to 1/4 second, with analysis on 25% of the pixels in the image rather than the default 12%. This should greatly reduce the chance that motion detection is NOT triggered even though a plate was read, but does not specifically guarantee that we will record an image when a plate is read.

If we want to logically link plate detection such that they trigger video recording of the event (makes sense), it's best not to rely 100% on motion detection. So we create a manual event, and configure the camera to record also when this manual event is fired. We then configure the plate lists such that whenever a plate matches any of the lists (including the default "unlisted" list), our manual event is fired, triggering recording of the event.

In theory you could actually disable motion detection and rely on the LPR events to trigger recording, but then if LPR is ever missing plates or not running, then cars would pass by without also being recorded.

The last 20-30 plates look like the video was recorded as expected and the bounding box is spot on. If you still need some help please let us know.

(7)
UI
Undisclosed Integrator #1
Oct 20, 2016

Thank you for looking at this for me. I did ask support about setting up a manual motion detection but was told this was not needed.... I will let it run over night and ask the customer to check data and recordings.

I wish all milestone support was as helpful as you Joshua. Why has it taken me to put something on IPVM to get help??

(1)
Avatar
Josh Hendricks
Oct 20, 2016
Milestone Systems

Assuming it is actually solved now, it's difficult to say why it has taken so long. Your opening description of the case actually contained the solution - no motion was detected during the periods of the missing video. Sometimes troubleshooting can lead down the wrong path, and it can take too long to discover that. I suppose that's what happened here.

In any case, I'll pass it on to our EMEA team leads to use as coaching material to improve our handling in the future. I've been with Milestone over 10 years now and I still learn something new every day!

(2)
UI
Undisclosed Integrator #3
Oct 20, 2016

Josh - you're a real credit to Milestone support. Now all we need is your direct line ;)

(1)
(2)
U
Undisclosed #2
Oct 20, 2016
IPVMU Certified

Wow! Nice fix, (hopefully).

You hardware guru, you ;)

(1)
Avatar
Simon Barnes
Oct 20, 2016

Well done! Could you have a look at the mobile app not getting a shutdown notifications as mentioned below. It would appear we both have the same problem and ongoing cases.

Avatar
Simon Barnes
Oct 20, 2016

Yes. I have found this a few times its as if they cant see anything obvious so they blame your kit.

I've got a problem at the moment where if a server shuts down you don't get any notification on your mobile app telling you a server is offline (it used to) then only when you switch it back on do you get a useless notification that tells you "a server that was offline is now back online".

This happens on all my customer systems that i've tested and i've tested quite a few so its clearly a global issue and not a server issue but Milestone want me to send them a load of info logs etc from a system. like i said its as if there stuck and they just throw the ball back in your court.

(1)
UI
Undisclosed Integrator #1
Oct 20, 2016

I am also finding the same problem on a number of my sites. I logged it with Milestone and was told its a hardware problem..... If I got a pound for every time they say that to me I would be a RICH man!

(1)
Avatar
Simon Barnes
Oct 20, 2016

well if you've got that problem and i've got the same problem that pretty much confirms that its their end we cant both have the same faulty hardware!

Avatar
Josh Hendricks
Oct 20, 2016
Milestone Systems

Thanks Simon and UI1, it's the first I've heard about this issue myself but it sounds like a piece of cake to reproduce. I'll get one of my team to have a gander later today and assuming we can reproduce it, we'll pass this on to our mobile/web services team.

I will also pass on the general comments about the perception that we are "always" blaming hardware. It is super frustrating to get bounced back and forth between companies. As such I encourage my own team to always be able to demonstrate the issue is not software - usually by taking it as far as to prove it is hardware - Wireshark, disk/ntfs errors, firmware discrepancy, something that can be taken to the manufacturer, or network admin to action upon. I'll voice your concerns with my peers and hopefully reverse that trend.

(3)
Avatar
Simon Barnes
Oct 20, 2016

Thanks Joshua my case number for details is MSC215297

MS
Max Scott
Oct 20, 2016
IPVMU Certified

I had some recording issues on a Milestone system and that happened to be one of the answers I received, but that didn't fix my problem. I called back later that day and ended up speaking with a tech support associate who heard my description of the problem and asked follow-up questions to isolate the issue.

Turns out that the desktop clients for some folks' stations were streaming the cameras at a lower quality than what was being recorded (to reduce resource load on the view stations). The problem was that the server then had to use loads of CPU cycles to re-encode the video streams to a lower quality.

The solution was pretty straightforward. Either don't have 150 cameras streaming all at once, or have them only show video on motion trigger (thus reducing the amount of data streaming at one time). Returning client video quality to the default amount reduced the CPU load on the server down to 10%, and the recording gaps stopped happening.

Milestone does seem to be more hardware intensive than other products that I'm used to working with, but this certainly was a learning experience. Despite my first call being of little help, my experience with tech support was pretty pleasant. In the future I'll be sure to overshoot specs on builds and keep an eye out for CPU levels.

(2)
UI
Undisclosed Integrator #3
Oct 21, 2016

Had the same challenge a couple years back. Milestone imageserver.exe will torch any server you toss at it if you have many high resolution cameras up at once and scaling back the resolution in smart client. On paper throttling back at the client looks good, kind of a HDSM-like option. In practice having too many cameras up will just burn cycles somewhere, whether at the server or workstation, regardless of what is set.

Specs on my old CPU cycle eating challenge:

~400 cameras with resolution from 2 MP to 5 MP

4 servers capturing roughly 100 cameras each

Milestone 2014 Enterprise Master/Slave setup

Windows Server 2012 R2

Dual E5-2620 V3 (12 total cores, 24 threads)

32GB RAM

2 workstations:

E5-1650 V3 (6 Cores, 12 threads)

16 GB RAM

Dual Nvidia Quadro driving 4 1080p displays each w/screens overly full of high resolution cameras displaying thumbnail size.

Try RDPing into a site where the customer complains about the $6k workstation being slow when the CPU is pegged at 100%.

Lessons to be learned: who watches 200 cameras at once and hopes to see anything? Also, the above challenges may have changed in some of the 2016 releases with Intel QuickSync support. Finally, never use master/slave setup on sizable sites!

(2)
Avatar
Mike Dotson
Oct 24, 2016
Formerly of Seneca • IPVMU Certified

Max,

You mentioned.."Turns out that the desktop clients for some folks' stations were streaming the cameras at a lower quality than what was being recorded (to reduce resource load on the view stations). The problem was that the server then had to use loads of CPU cycles to re-encode the video streams to a lower quality. "

I saw this in my lab some time ago during testing and discovered that the only thing (at the time) that you 'could' alter without kicking in the transcoding operation was the FPS. The SmartClient docs mention this in a subtle way (see pic).

One also has to be aware of the server loads that the Mobile Server and Web Client can make.

I recall one doc mentioning that for a Mobile Server...once you got past 10 cams...to install it on its own machine.

SmartClient doc

UI
Undisclosed Integrator #4
Oct 24, 2016

Yep! It's always the Device Pack is not up to date. Even when a new Device Pack is installed, it rarely corrects any major issues.

Or it's a problem with the network.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions