I would blame the manufacturer more than anything, but the integrator shares a portion, too, albeit smaller. I say this because I have always seen manufacturers oversell analytics. So when manufacturers do this, isn't it logical the integrator will too, since they are relying on their manufacturer "partner" to give them reliable information...? But the integrator shares blame too for "trusting but not verifying" the claims and doing their own R&D to test it's limitations. Testing products costs time & money, but that is the nature of the beast, it just is what it is. We can blame the "CSI Effect" all day, but if we sell expectations based on sales literature and not properly vetted one, because a competitor is more willing to sell unrealistic performance without doing proper testing and so therefore at cheaper cost, that is on us (the integrator).
Luis, I do not know the specific role of integrators or consultants in this case.
That said, in general, I do agree that integrators hold responsibility. Indeed, I think the risk is greater for the integrator since the integrator is on the hook for labor to make it work. I have seen many projects where the integrator lost substantially trying to make analytics work over and over again.
You're right. I made assumptions based on limited info, because that is how it goes most times, but doesn't mean all the time. For all I know, maybe the analytics were hard spec'd because of work done by the manufacturer's sales reps and the integrator had no say. I only mentioned integrators because I have seen it happen from that side, too.
It really is this simple. Either you essentially eliminate all false alerts or you fail.
Doesn't it depend on the application?
For instance, a cross-line detection in the metro might be annoying if falsely triggered once a day, esp. without 24/7 live monitoring. On the other hand, once a day false alerts for cross-line in the Capitol building might be an acceptable trade off considering the assets at stake and the budget available.
Which brings another point, false positives can often be significantly reduced by lowering the sensitivity, which of course will increase the false negatives.
So do you mean "Either you essentially eliminate all false alerts without missing ANY real ones, or you fail?"
I think that slider is vastly different depending on the circumstance.
As an example, 1 out of 3 times I travel thru the airport the detector flags me, without apparent fault of my own. They can't be overjoyed with this result, but I'm guessing they could also turn down the sensitivity if they wished less false positives.
"An alarm box near the front entrance of the White House designed to alert guards to an intruder had been muted at what officers believed was a request of the usher’s office"
The alarm had apparently been regularly generating false alerts, annoying staff.
That said, I do agree with implied fundamental point. Situations with higher risk profiles are going to tolerate more false alerts.
But, as the White House incident and my own experience with military video analytic deployments shows, it is not as high as one might imagine.
1000 cross-line boundaries per day is 1 every 1.5 minutes. Most staff, even at critical infrastructure sites, would turn it off, unless you were at very high alert for imminent attack. Operationally, floods of false alerts are debilitating.
As a business, though, it is still hard to find and sell analytics if you are really generating 1,000 false alerts a day. You surely will get some but at great pain for your sales and field engineering team.
Conditions in metro are really bad for this kind of analytics when people are staying and waiting without any movement.
Is the OP suggesting that people standing still are causing the nuisance alerts? If so, a size limitation on the analytic may reduce them. A target would only count as an object if it was smaller than a person. (Which opens up the problem of really small persons)
Even so, a train platform is not a good application of "object left behind." People naturally set things down while waiting. Neither is the baggage claim area at an airport, frequently cited in news reports regarding the "magic" of analytics.
Are the 5-6 alerts per hour truly "false" (mistakes by the analytic) -- or "nuisance" (technically correct but not what the customer wants to know)? As I have said in this space before regarding analytics -- often what the customer wants to know, they don't want to know. A screen shot of one of the "false" alerts would facilitate discussion.
It really is this simple. Either you essentially eliminate all false alerts or you fail.
A project can fail even with perfect analytic performance. This may be such an example. In addition to accuracy, I still contend that managing customer expectations is essential to success with video analytics. Based upon the OP information, IMO, the customer's expectation in this application should have been, "This ain't gonna work."
This article is bang on. Unless video analytics are perfect, you just cannot sell them.
Countless times we are asked to set up email notifications for Video Motion Detection, and in 100% of cases we have ended up turning it off for the customer due to high number of false alarms. Forget line crossing, facial recognition, etc, this is basic VMD. It absolutely does not matter which brand or product you use, or how much tweaking/adjustments you make, you will get false alarms. Unless you turn it down to be so insensitive that it does nothing.
Sadly, video analytics are unlikely to improve while lawyered-up companies like Avigilon are hoarding patents, so we are stuck here for a while.
Unless video analytics are perfect, you just cannot sell them.
I tend to disagree, but I think you need to set proper expectations up front, and know when to tell a customer that their expectations cannot be reasonably met.
Forget line crossing, facial recognition, etc, this is basic VMD.
I understand what you are saying, but I think you may be drawing the wrong connection there.
Video analytics (for perimeter protection/notification use cases) is not about what you DETECT, it's about what you IGNORE. As your VMD experience shows, it is easy to send an alert on any random movement, that is not very valuable. What you want is to ignore more things, without also ignoring people (or vehicles if that is what you are alerting on). Being able to intelligently ignore things is where you move from "VMD" to "Analytics" (IMO).
VMD is very useful for managing storage when the customer's requirements do not mandate 24/7 recording, no need to record a generally static scene. VMD does not need to be super accurate, it is OK if we end up recording too much, within reason of course. But VMD is not good for sending alarms out unless you have a very sterile environment, like inside an office, because it will trip on blobs of pixel change without any care about what caused the pixel change.
Line-crossing applications (though I consider those a somewhat weak form of analytics in the current market) can and do yield better results than VMD, if you have the right product and the right application.
It absolutely does not matter which brand or product you use, or how much tweaking/adjustments you make, you will get false alarms.
I 100% agree with this, zero false alarms is not achievable in the sense of making a general claim. It is a matter of WHEN not IF that you get a false alarm (or 20) from any system. However, if the false alarm rate can be kept low, and the event handling side of the alarm (the software/VMS interface) makes it easy for the operator to dismiss these you can have a system that is very "sellable" and has a good ROI for the customer.
Part of determining where video analytics for alarming is worthwhile or not involves determining the overall value of the customer's problem. If you want to protect a cherished garden statue in your yard, the cost of setting up the system, and the inevitable false alarms will likely outweigh the value of your prized gnome. If you have $50,000 worth of inventory in an outdoor store yard and are getting hit for $1,000/month in losses you can likely find a solution that is cost effective.
It still comes down to choosing "No False Alarms" or "No Missed Events" though. If you want to reduce false alarms to near zero you are inevitably going to have to set sensitivity or selectivity to the point that you will end up missing some valid events (and thereby likely have losses that may have been preventable). If you want no missed events, then you are going to get some false alarms as well.
Brian, this sounds like a very long-winded way of agreeing with me and proving the point of the original article. Analytics DO NOT WORK. You choose between many false alarms or many missed events. That means they DO NOT WORK.
Like the article says, it doesn't matter what you the integrator or manufacturer thinks is acceptable - it matters what the customer thinks. And the example about how ADT would go out of business if they had daily false alarms perfectly illustrates this.
How do you define "WORKS" and "DOES NOT WORK" in a broader sense?
If your customer wants a camera able of capturing license plate details in a parking lot, and you install such a camera while also explaining to them that the camera will not be able to capture those details from 11PM-6AM because the lot lights are on a timer would that system "work" or "not work"?
What if a car speeds through that parking lot at 50MPH and the plate is too blurry to read? Is the system working, or not working?
I do not think you have to settle for "many false alarms" or "many missed events" as a general statement. I believe there are certain products that are far more likely to generate false alarms (or miss events) than others, and I believe there are also particular scenes or settings that are more prone to false alarms. But I also believe that there are analytics products on the market that can cost effectively satisfy a wide array of customers and applications.
The specification determines what works and does not work. If the specification stated that LPR needs to work 24/7 then the timer needs to change and/or alternative lighting needs to be provided. This is the job of the intergrator to understand the specification and know the site conditions after a walk thru.
Similarly if the specification calls for LPR at all speeds the integrator needs to take exception, or, require installation of speed bumps to slow the car down, if he is to meet the specification.
The issue spans multiple manufacturers, and in fact spans multiple industries. In the automated fingerprint identification system world, I have seen RFPs that request 100% accuracy. Not near perfection, but perfection. Such a requirement does not get a response of a simple "Comply" - primarily because if a vendor did state full compliance, and that response made it into a contract, there would be trouble down the road.
I'm first to completely agree on all statements. But this is not manufacturers fault! We always warn our integrators that real time analytics is not perfect, but they want the project and that is all!
They are even doing pilot installations, but trying to hide real results. So integrators often knows the real situation, but they prefer not to say "not" to customer, because in this case customer will find another one who will say "yes".
I don't know the right behavior for manufacturer in this situation. We have this analytics, because everybody has and often it is requested. But we know, that to manage effectively you need operators to confirm often false alerts and we always saying this. Another our decision is to set zero price for online analytics! We have all kind of online analytics and it is free! So, we are not giving the reason to complain, that online analytics is not perfect. Yes, it is not perfect, that is why we are not taking money for it! Now we take money only for forensic search analytics as this is really helpful tool which increases efficiency of investigations, saving operators time and money as result!
Let's dissect this first statement : "But this is not manufacturers fault! We always warn our integrators that real time analytics is not perfect, but they want the project and that is all!"
1. this is not manufacturers fault
2. We always warn our integrators that real time analytics is not perfect
So the manufacturer is not at fault because they warn customers ahead of time that their product is faulty. This statement is a perfect example of both why nobody uses analytics and why nobody is willing to pay for analytics. It completely proves that the original article was bang on.
"So the manufacturer is not at fault because they warn customers ahead of time that their product is faulty."
Faulty would be if it did not work as designed or expected. (The car's brakes fail.) If it works as designed, but with disclosed limitations, that is not exactly faulty. (The car does not brake the exact same stopping distance every time.)
To extend your analogy, your brand new Toyota's brakes work flawlessly when you are driving inside a warehouse with optimal lighting, but when you take the car down the highway on a mixed sun/cloud day, the shadows created from the clouds cause the car to brake suddenly every 30 seconds or so.
So you return to the Toyota dealership and complain. The sale manager says you need to manage your expectations as a customer and also assigns partial blame to the salesperson for not fully explaining that the brakes only work in optimal conditions.
So you avoid driving on cloudy days completely since the car is basically unusable. But then one day you take it out in the rain..
John, as an engineer who has been involved with "video analytics" for over two decades I am amazed by the strides the industry has made in its algorithms, but, at the same time, truly amazed by of the markets perception of video analytics capabilities.
My analytics background comes from the semiconductor industry where we had to commit to 100% identification of true defects and false alarm rates close to zero percent. This was not an easy task to achieve. It is important to understand that our task was easier a surveillance application since our scene was always the same and the lighting was always the same. With surveillance applications you have scenes which are constantly changing: people/cars moving, trees waving, day/night transitions, weather implications etc.
I believe the problem lies with the manufacturers not really explaining what video analytics can do and under what conditions they can do them. Although I do not know the specific details my initial reaction is that arming 1000 cameras within a metropolitan area was irresponsible. The manufacture must have known of the application, the extent of the installation and the risks associated with such an installation. To not expect 5000 false alerts an hour and not know this would be unacceptable is borderline criminal.
My view is that analytics, even in their current state, have a place in our industry. Applications which have controlled environments or applications where the identification of an alert is far more important than the false alarm rates, are examples where analytics can be deployed. For instance, if the manufacturer told the metro center to deploy the analytics in 4 or 5 critical areas, educated him on what to expect for a false alert rate, the customer would probably be seeing analytics as a success.
Aspirin does not always work to take the pain away- faulty, pull them off the market?
Aribags don't always save lives- faulty, pull them off the market?
Speed limit signs don't always make people travel the speed limit- faulty, pull them off the market?
CAT scans don't always catch a blood clot- faulty, pull them off the market?
What are we arguing about? That analytics are not 100% accurate so they are faulty and never be used? Even if you explain the limitations to a customer because they will ignore what you said, or is in writing, and expect 100% accurate results?
Or whether or not the customer was told what to expect?
None of your analogies are applicable here. I could counter them all with equally ridiculous arguments.
#1 - Aspirin has a huge market, because there is a decent chance of relieving pain. Analytics have basically zero market, because using them has a high chance of causing pain. Maybe there is a related conspiracy theory here.
#2 - Airbags don't accidentally deploy when you drive down the road on a cloudy day, or the lights from another vehicle reflect off a wall and into your windshield. If they did, people would improve airbags or stop using them.
#3 - Speed limit signs are informational, and allow drivers to know valuable information. Imagine if speed limit signs were placed every 25 feet on some roads, then other roads you just never saw one at all, and then on cloudy days the signs all changed to show the speed limit was double, then changed back intermittently as sunlight reflected off them. How useful would the speed limit signs be then?
#4 - CAT scans are a valuable tool because they might reveal a blood clot. Imagine if CAT scans falsely revealed blood clots because it was raining out, or the bushes outside the hospital were blowing fiercely in the wind.
There's nothing really to argue here. Video analytics do not work well enough to be considered useful. Own own experience certainly confirms this, as does the video analytics market (or lack thereof) and of course the original article which was bang on, plus the contradictory comments from that one guy.
A better way to win this argument would be to cite a useable analytics package (and then lose the argument when you run an IPVM test on it to show how it actually doesn't work after all.)
Video analytics do not work well enough to be considered useful.
Such a blanket statement.
Do you really think no one finds VMD alerts useful? Inside a store during the early A.M. for instance?
Again video analytics don't have to be perfect or even near perfect to be useful.
They are still useful to SOME people in SOME cases.
Example: I have a new car with "Video Analytics" with Blinker activated change lane warning. I would say 2 out of the ten times it warns because the road curves ahead and it wrongly thinks a car is in my path. Many times it has stopped me from even beginning to change lanes when there is another car there.
You are ignoring this particular customers perspective in this situation. The statement was made that there are 5000 false alerts per hour (over 1000 cameras). How can any municipality respond to 5000 alerts an hour? There is such much "NOISE" that the "SIGNAL" (true alert) is going to get lost. It is our jobs as manufacturers to provide meaningful information and to separate the signal from the noise.
Working with pixel based analytics has to be much more difficult to master than the PIR based motion sensors that ADT uses so not sure if thats a fair comparison. I somewhat agree with the manufacturer that customers expectations are too high but I do think this should have been explained more upfront before the sell was made.
Im a big fan of the cameras that have built in PIR though. Its much more of a true motion event for sure. I'd like to see more of them. The only issue is if the user wants to detect motion from several hundred feet away, then thats where PIR's start to lose effectiveness. But I'd like to see more advancements in this area (cameras with built in PIR's). If I were a camera manufacturer, I would do this. Not sure if PIR's could help with advanced analytics past typical motion detection though.
Working with pixel based analytics has to be much more difficult to master than the PIR based motion sensors that ADT uses so not sure if thats a fair comparison.
Sean, for sure, on pure technology, PIRs are far more mature than pixel based video analytics.
For the comparison, I mean from the user's perspective of dealing with false alerts. The user may not know anything about the underlying technology but they do know what operationally they can handle. Both technologies send alerts about security risks, one technology is just a lot worse at meeting customer's expectations of valid notifications.
Yeah I agree. I think the solution would be to be more upfront with the customer about user-ability with these things. If reliability on video based alerts was a stated requirement on such a large job, there should have been much more clear communication about the false alarms that they "will" receive. We do the same thing when people ask us if there is email alert capabilities. We say "yeah, for your your outdoor cameras its going to be blowing up your inbox with false alarms" and then explain the whole video based motion detection thing.
I'm no technology manufacturing expert at all, but I just dont see how they would solve false alerts for cameras installed outdoors for pixel based analytics. I dont think it will ever get down to the same reliability as a PIR would have unless computing in this area comes a very long way. There are way too many things that can trigger false alerts: Light changes, Bugs, shadows etc.
If a person can tell whether something is a real alert just by looking at the pixels, then it's possible for the computer to evaluate by a similar heuristic.
Some are harder than others to emulate...
I know you know this but I do feel it's worth emphasizing. There's a widespread conception that video analytics 'heuristics' including understanding what people really 'look like' but it's typically far less sophisticated with very rudimentary inexact heuristics that a 3 year old would outclass, ergo the common false alerts.
I think we need to keep an open mind here. We can be somewhat dismissive of things like MJPEG and CCD's without too much fear of being surprised in the future, but with analytics, I think we have to really watch carefully as this unfolds.
There are likely to be a broad range of capabilities and price points mostly uncorrelated. Off topic: I would imagine within a couple years Analytics would warrant a permanent sub-section on IPVM.
I know you've heard it before; for the last decade (or more), you have heard the promise of game changing Analytics that are right around the corner.
So what I'm saying is that maybe there is no sharp corner, just a slow bend that we might miss if we don't look for the cumulative result in incremental changes.
I presume that the PIR triggers a preset in the PTZ??
These systems suffer the same problem with disconnected camera systems and alarm systems where what is triggering the event cannot guarantee that the camera view is less than 1 second within the event and can see in the field of view everything that might enter the detectors zone of coverage. If the monitoring station operator is uncertain what triggered the event, and they don't see what triggered the event, what will they do? Annoy the customer and ask what do you want us to do...
Are ipvm users aware of the Video Verification Best Practices as published at the www.ppvar.org site? This resource provides an excellent place to start in creating an appropriate SOP for the system design.
New York, fresh off one disaster with analytics asked a friend during a presentation how well his "object left behind" analytic would work. His answer was "see something, say something," like your wall posters would work better.
The board agreed and disclosed they use that question as a "BS detector".
While this subject is debating all analytics, one in particular "object left behind" was the focus of this issue.
What people want is "dangerous object arrived" and anything less than that is a problem for managing expectations. If the system accurately detects an article left behind like the backpack from the Boston Bomber that's good? Right?
How soon do you respond and with what? Should they have shut down the marathon because of a backpack? How many were set down that day for more than say.... 30 seconds?
If you want to assign blame, it's "all of the above" for not thinking through the whole scenario. It would be different in a sterile environment, back hallway door, secure facility....maybe, possibly, depending.
I happened to read U3M's comment Thursday morning, as reports of the situation at Joint Base Andrews were circulating.
While the situation isn't completely clarified, it appears that (a) the base was scheduled to hold an active shooter drill today, and (b) someone apparently reported someone who looked like an active shooter.
I know of no algorithm that can definitively conclude whether this was a threat or not. (For example, what if someone took advantage of the drill to pose as a fake active shooter, when in fact the person was a real active shooter?)
Analytics need a yard stick measurement or what happens is you get every man and his dog adding it in cameras etc and it fails. iLids is a uk test, I'm interested to see what standards exist for testing?
ILids is a uk test that measures an analytics products capability to pass numerous quite stringent tests under diverse conditions. The result is you get an analytics product that is 'approved' for say secondary perimeter detection. Its not perfect as a test scenario, but its all we've got at the moment. My issue with it is it merely indicates a product is capable of performing adequately under set conditions - manufacturers and integrator alike continue to install I'Lids certified product under scenarios that are so far divorced from the test conditions to make the certification worthless.
I spent 3 years field testing an ILids product certified for perimeter use on a CNI project. Unfortunately, the overall system design was so far away from the test conditions and had some fundamental flaws that made the analytics near useless. At the end of year 1 I did not believe that the product was capable of meeting the requirements and told the supplier so. By some fundamental changes by the integrator to their basic design, and extensive recoding by the analytics manufacturer, they managed to produce an end result that I would not have thought possible. However, this took extensive rework to basic system design elements plus extensive time on site by the manufacturer, changing configuration coding at a level far beyond the capabilities of an integrator. Yes it eventually worked to a degree, but the cost to get it there far far exceeded the budget in the project.
I don't consider analytics a product for mass market use in all but the most basic and controlled conditions.
The last time we reached out to i-LIDS 3.5 years ago, they would not share any information, saying:
There is a list of all the i-LIDS approved systems on the government security catalogue, however this is not a publically available due to the nature of the contents. All the manufactures who have been awarded the i-LIDS classification will have an i-LIDS approval seal on their products, this can be found publically.
This aspect struck me as strange. The only way I can find if a system is approved by the government is to ask the seller? Wouldn't a list directly from the government be more authoritative and less risky than asking a seller to vouch for themselves.
i-LIDS does not seem to get much recognition in North America so i-LIDS actions is more curiosity here.
The UK have recognized that iLids may need to be updated. They are leaning towards replacing it with new scenarios and a performance level versus a pass/fail test. I believe its more law enforcement focused- project VALE
Taking of performance testing, did any one attend the recent NIST workshop on video analytics http://www.nist.gov/itl/iad/2016-workshop-on-video-analytics-in-public-safety.cfm ? if so, what did you think?
2) TECHNICAL NOTE - Of the Video Analytics tasks - detecting abandoned objects in a busy place like a station is one of the very hardest. The level of machine vision required is up there with driverless cars driving around cities.
To understand why, look at the numbers. Let’s say each station has 50 cameras and each camera 'sees' 20,000 people per day. Every one of these video 'events' needs to be analysed to see if anything has been left behind and whether it might be a threat. During some periods of the day the cameras can hardly see the ground at all, it’s just full of people. And the user wants less than one false alarm per day, which is one false alarm per 1 million events.
Video Analytics can deliver 95% accuracy or maybe even 99% accuracy. But looking for 'rare events' in large quantities of data as in this case requires 99.9999% accuracy. No Video Analytics system on the market comes anywhere near this yet.
Unrelated to the topic, but directly related to your post: Try before you buy is impractical for many systems because the greatest cost is the time and training necessary to attain competence within such systems. Lets hope that, for complex systems, there are effective metrics for suitability that are much less expensive than a "try before you buy" investment.
I think Video Analyitcs in its current format is totally wrong, until some Artificial Intelligence is not developed to make proper decisions regarding an event.
For a human or an AI, its not a hard job to decide that someone REALLY left a backpack on the floor which he carried before, but an algorithm can only see pixel changes on a static image which it tries to analyze and cannot detect such a simple thing for example in a crowded hall where people are constantly moving in front of and behind each other.
Instead of looking for left objects, they should employ well trained dogs, who can scent explosives and get the attacker before they enter the area. A dog can be a lot smarter than any camera with analytics...
The cost of dogs and humans exceeds the cost of analytics in OPEX so the poster and using passengers was the next best thing and random human/dog sweeps.
Object Left Behind also morphed into Object Separation which then set a whole new question of alert timing.
I believe analytics should be scenario and response based.
Requirement...guy hops fence, detect him and ignore all other events such as bugs, birds, shadows, flashing lights, storms, lightning, fog, rain, snow, wind(shaking of camera and objects in view) darkness, sun facing and blinding. That's an easy one for most analytics in the upper cost range.
Here's a hard one.
Requirement...bomb left in train station platform and ignore all of above as applies, and at times it does, and add a timer because this will happen and sometimes I want an alert and sometimes I don't.
ILids was an easy pass. You have the videos and you get to tweak and manipulate until you run the video for the test. It weeded out some analytics I'm sure, but the cost was high and difficulty was low to medium.
The last time I asked only 3 companies were approved for iLids 'sterile zone' and 1 for 'long range thermal imager.' I don't know of any company who passed the others 'left baggage'/ 'multiple camera tracking' etc.
One of the issues is that people often don't want to invest in testing analytics prior to a sale ( the sales man might not offer it due to cost).
I would be nice if you could have a centralised open source database of videos that represent a certain scenarios and analytics companies tested themselves against them
We report that video analytics work poorly and members effectively respond 'Of course, we knew that, why waste time on this?"
To make it more challenging, video analytics testing (done right) takes a lot of time (a lot more than cameras since so many more factors impact analytics).
So our issue is that few seem to work well, members do not seem all that interested (especially in ones that perform badly, which they expect) and it takes us a lot of time to do.
We are still going to test video analytics (Agent VI is up next, whenever Avigilon ships Appearance Search, we will test that, Bosch improved analytics is something in queue as well, etc.) but until we see a real uptrend, video analytics testing will remain a lower priority for us.
What made iLids workable when I worked with it was the time spent to create scenarios that all analytics would be pushed through. Back then it was CD's. As a manufacturer you were provided sample clips to tune with but the full test was not disclosed. Similar to an SAT practice test.
There was a US version of this type of testing with Sandia Labs that was on video tapes for peimeter fence line only.
I wonder how iLids manages the camera built in analytics? Aim the camera at a monitor and play the DVD?
Its not practical to attempt to run the exact same tests over and over both cost and comparison wise.
I wonder how iLids manages the camera built in analytics?
It shouldn't change things much, at the conceptual level.
You would have to get the clips into folders on the camera. Then using the camera's edge recording/playback, (and assuming that the mfr allows their analytic to run on recorded video), you would run it locally.
Everything is straightforward at the conceptual level.
You would have to get the clips into folders on the camera. Then using the camera's edge recording/playback, (and assuming that the mfr allows their analytic to run on recorded video), you would run it locally.
Never heard of a camera allowing this, positive it cannot be common.
To your point, I am sure it's 'conceptually' possible but it is not something that is normally allowed.
What I was suggesting was instead of IPVM performing the tests directly, as this is cost and time prohibitive, that IPVM determine the most typical use cases and create or crowd-source a series of progressively more difficult video footage along that same scenario.
For example, cross line detection along a fence.
Level 1 : Empty Scene, great camera positioning, object larger and closer to the camera and thus easier for VA to detect and filter out nuisance alarms.
Level 2 : Same as before but object farther away and thus smaller (i.e. represented by smaller pixel density)
Level 3: Scenario that introduces shadows, or a scene where the clouds are shifting and the sun shines through causing a massive change throughout the entire scene.
Level 4, 5, 6, 7, 8. progressively get more and more complex.
This way over the next 8 years we can see if any VA manufacturers want to take a go at the challenge and we can then discuss their results.
You could make it even more challenging if a VA mfg can produce a result across all levels using the same configuration and without significant tuning of settings per each challenge.
This way the burden of the initial testing would be on the various manufacturers and if after they have succeeded at a number of scenarios and difficulty levels, only then would IPVM setup an independent testing to validate their submissions.
Nice article, John. All the comments are very good. It's not as black and white as some demand though. Everything depends on where and what you are trying to use the analytics for. We have struggled with these issues for a long time. Given the infinitely variable environmental conditions, I don't think near perfect performance is a realistic expectation.
This is reminiscent of discussions I used to have with customers over volumetrics many year ago. Even now, 98% are still nuisance activations.
We hope that the intelligence in real-time video analytics will make it an effective tool for many applications. But, many on both sides try to turn their hope into expectations.
A minimum level of false activation gives a human an opportunity to discern what is true short of numbing the system to the point that you miss a real event. But too many falses desensitize reactions rendering the system ineffective. Same old dilemma.
I'll keep hoping, but I always temper my customer's expectations with reality.
Forensic analytics have a brighter future than their real-time cousin.
Forensic analytics have a brighter future than their real-time cousin.
Dale, nice contrast!
I agree that forensic analytics have greater potential to actually work / solve problems.
The challenge though is that forensic analytics generally have lower value than real time analytics, e.g., alert me when the wanted criminal comes into the mall is generally more valuable than tell me if the wanted criminal came into the mall last Thursday.
End users tend to be willing to pay a lot more for real time alerts (given the potential value they bring) but then they do not work as well often, ergo the dilemma because forensics and alerting.
While we wait for the perfect capture rate, you must plan resources for human discretion as well as response. Some organizations aren't ready for the responding resources they need for alerts. Sometimes it is better to not know about a condition/event than to know and not respond appropriately. Hopefully, the mall cop won't shoot until they have a positive ID so I think human assessment/intervention will be around a while. At least until we reach "Minority Report" level. Even license plate recognition isn't 100%.
|"forensic analytics generally have lower value than real time analytics"
Behavioural and Traffic analysis are examples of forensics that are extremely valuable in planning everything from roads to emergency evacuation to product/service placement.
Also, there are a lot of other practical applications for real-time analytics that are less critical than IDing a criminal. Blocked exits, line queues, spills, loitering, object removed, speeding, tailgating, and I'm sure many more.
Would be interested in knowing "exactly" the application alluded to in the "left behind" or metro analytics situation. It also appears that there is confusion of what is meant by false positive. A false positive is not an event generated where it is not a threat. A false positive is where what the analytics should ignore (eg. birds, tree branches moving, animals) is generating events. The analytics did what it was supposed to do if it is detecting people - what was the error is not considering the frequency with which non threat events will be generated by normal activity in the scene.
A common customer request is dealing with crime in their parking lot (eg. golf course parking, apartment parking, metro parking, casino parking). These applications are not workable with analytics for many reasons that has nothing to do with analytics; while analytic applications involving car dealerships do. I run into these kind of installations from time to time - all have been turned off. If your staff doesn't understand why they don't/can't work you will spin your wheels in misapplying video analytics.
Likely there are similar principals that make a left behind or metro analytics situation workable or unworkable.
I'd ask for a bit more clarification on the user's definition of "false alert".
For example If the analytic is set to notify on an "abandoned object" and the dwell to alert is 1 minute once an object is detected and a passenger sits a suitcase down for 1 minute is it a "false alert" if the analytic triggers? By the customers definition it is, but is it really since the analytic has no way of distinguishing that?
That may be an oversimplification, but again, more details are needed.
I'm in agreement with those that stated it was a bad application of the technology for this scenario.
By the same token, we also need to examine the customers motivation for wanting it to begin with. He has 1,000 cameras. How big is the staff that monitors 1,000 cameras? Nothing takes the place of actual human surveillance and decision making, but obviously there's no way the customer can afford staff to effectively monitor it so he's turned to technology as a force multiplier.
I'm hard pressed to understand why an initial pilot of the analytic on a dozen or so cams wasn't done to begin with so he could establish a baseline to determine if he could live with the "false positives" (if that is indeed the case) to begin with.
I would like to join this conversation from a sensor POV. We have been testing, with success, the combination of sensors working as the trigger for video analytics (we call this a IP video double knock internally).
Testing the complementary features between the video analytics and physical sensor in outdoor environments, we have set up a live system using both technologies: Optex sensors and video analytics. The footage gathered gives a good insight on how the double knock can decrease false alarms and provide real time forensic information:
Introducing physical sensors solely dedicated to the function of sensing and independent of the camera view and angle can help. Using different sensing technologies that can suit different environments also help lower false alarms. Sensors can also help cameras eliminate cause of false alarm in outdoor environment where lighting conditions will hurt video analytics.
How does it all apply?
Sensors can be used to trigger a number of devices including non video devices such as lighting, signaling systems, barriers, fogging systems. Sensors can detect in areas where no camera can be installed: walls or network cable. Sensors can cover detection areas difficult to achieved with a camera. Sensors complement and validate video based detection to create a “double knock” system.
All in all, there is not one technology that can do it "all." Why not use multiple technology strengths based on environmental, system configuration and requirements? I hope this POV can help manage some of these missed expectations.
There is certainly room for improvement, and here is an example; In automotive-grade analytics used for self driving cars (ADAS) the failure rate requirements are extreme. This level of analytics deploys a lot of processing performance, multiple sensor types, and in general use other classes of algorithms than what is typically used in security today. True, the use cases as different, but there is significant overlap in tracking and object classification.
A challenge in surveillance is the inability to charge (enough) for analytics in a smallish market -> which means less R&D can be invested -> resulting in less performing solutions, etc. Not so in automotive analytics. That is a much bigger market and good analytics is a must with tough industry conformance testing, which drives specialized chipset and algorithm development. My guess is that such automotive technology will become more affordable and trickle down to surveillance in next few years in some form, making security analytics more reliable and usable. Security being a relatively small market, has great traditions in deploying standards and technology from other bigger markets. The era of seeing cameras and robotics has begun.
Yes, the era of seeing cameras has begun, but not automotive industry is going to help. Automotive industry is always far behind IT and general electronics. The major limit for perfect analytics is not ideas or algorithms. Imagine, that mathematical model of brain is already designed and open sourced. The only problem was calculation performance and education. NVidia already designed platform and SDK for deep learning (technology to educate this "brain"), so we now have a platform. The only challenge is to find the most effective ways to teach artificial security guards :)
For what its worth, every implementation of abandoned objects or removed objects using VMD (video motion detection) is destined to fail.
I do however have first hand experience with more than one implementation of abandoned objects (or removed objects) using NMD (non-motion detection) with a video analytics company out of Toronto circa 2008. I wont plug their name, but you can google Non-Motion Detection as its a patented technology which allows them to operate extremely effectively in crowded environments.
With standard VMD implementation the detection times for abandoned object is often within seconds of the object being placed. This is because of limitations of this solution using VMD.
Simply ask any analytics company if their object detection is based on VMD or NMD and/or specifically inquire whether their solution can detect an object that is left behind for 5 minutes in a busy environment , i'm certain they would hesitate and be vague in their response. However, if you ask the video analytics provider that's capable of using NMD, i'm certain you'll get a much more confident response.
Better question for the sales people are to ask if they can detect a car parked for longer than 20 minutes. Its guaranteed that VMD will fail in this scenario, but for NMD this is where it would shine.
Its a pity that RFQs arent more specific. Customers should request that a system can detect objects left behind for longer than 3 minutes or 5 minutes. It would protect themselves.
FYI, another highly effective application of NMD is graffiti detection.
Sean, I'm sure that no one is using VMD for adandonded object. This is simply impossible to have any result with VMD, because we are looking not for moving objects. Calling it NMD is like salesmans trick to highlight your uniqueness to customer who doesn't understand anything in analytics. Typical abandoned object analytics idea is to summarize images for long period of time to predict the background and summarize for shorter time (20min for your example with parked car) and than compare this too and difference will show the abandoned objects. I never ever heard about using VMD, because it is just sily.
You are certainly right about it being impossible to have any result with VMD, however, if you've ever experienced the snake oil salesman of Video Analytics I think you would reconsider your first statement about being "sure that no one is using VMD for abandoned object" :)
I hope that readers don't take your statement and assume all implementations are the same.
I've had experience years ago with VMD based attempts at Object Left behind. They only ever worked with a short duration and relatively empty scenes.
For the sake of users like the "Manufacturer President", that prompted this discussion, I thought I would share my experience in hopes that integrators and consultants can become more educated and discerning. I hope that they ask the right questions if they ever are to offer this to a client. If anyone is to take away anything from my comments is to ask about the implementation, ask about a busy environment and particularly use of longer durations. This will help the integrator filter out solutions they should stay away from.
Ultimately the best advise is to pilot/demo any system within the proposed scenario.
...I think you would reconsider your first statement about being "sure that no one is using VMD for abandoned object" :)
Since your entire post was dedicated to disparaging VMD based object left behind analytics, where the only stated alternative was the patented NMD technology, I would assume that the VMD form are quite common, if not the norm.
Can you help us by indicating what products that you are familiar with that use VMD for abandoned objects?
I usually give up on reading comments after 8, maybe 9.
Admittedly this subject is close to my heart and I have played the devil’s advocate on many discussions such as this one in the past. So I’ll try to be objective here, as much as I can after 11 years in Video Analytics and 4 different manufacturers.
I will work with the main story and avoid response to individual comments as there are too many of them.
“Conditions in metro are really bad for this kind of analytics”
Well, no freak’n way, Einstein!! BUT YOU REALLY WANTED TO CLOSE THE SALE, DIDN’T YOU?
I can write a book about all those projects I have turned down, in which VA was just NOT the rights solution!
From detecting sharks on a Florida beach front hotel, to determine a friend from a foe (U.S military…), VA is not meant for that!
Nevertheless, saying that video analytics is a worthless solution?? It is like saying that police radar detectors are useless, because of false alarms...
It is true that very few (If any) VA companies made it to the finish line of profitability. We can blame many factors such as timing, investors patience (or lack thereof) and bad leadership.
Nevertheless, we can’t take away some great success stories in which Video Analytics are used successfully and almost exclusively. One example is the Remote Video Monitoring business. There are tens if not hundreds of small time operations that cater to 100+ locations and provide them with useful service.
“Does an ADT burglar alarm false alert 5-6 times an hour? No. Do you think ADT would be out of business if it did? Yes.”
First off, the burglar alarm industry, for the longest time prior to recent years regulations cost the tax payers $2B annually in false alarms. Statistics don’t lie and 90% of all police dispatches were false alarms! In fact, the reason why VA is so useful in the RVM business is recent requirements for Video Verification of alarm events.
In my past 11 years in the business, there has been very little improvement to VA performance. I’d argue that we even went backwards on innovation.
I don't know, maybe greed, made up market value projections, and down the loophole of inexcusably dishonest marketing campaigns. I have recently gave up my job due to that exactly.
We will never eliminate ALL false alerts. Heck, the radar detector in my car drives me nuts, but it saved my ass on countless occasions!
For fence-line intrusion detection, where the alternatives are "dumb" cameras, PIR triggers camera, or fence sensor triggers camera, Analytics is still the best option. With current thermal camera prices, thermal analytics is more cost effective than most solutions and provides the best combination of detection/verification.
Bottom line, for perimeter detection, what is a better alternative??
As to "Object Left Behind", a requirement to detect potential bombs left to detonate in a crowded mall, subway station, etc. From a pure optics standpoint, will never be a reliable solution. those environments are too crowded. in most cases the object will be blocked from camera FOV. Where it will work, indoor, sterile environment, it beats the purpose..
As to the guy with the new bombastic acronym, if you wait 5 minutes or more to provide a detection alert, the bomb already went off..
The ONLY current solution for suspected objects and suicide bombers is human vigilance.