We tested AnyVision back this past winter. My bosses were quite impressed with it, as was I but I had no previous experience with Facial Recognition.
We only had about five employees on the watchlist, and then a series of known actual subjects (20, I think?). Tried a set of identical male twins. It seemed fairly accurate, though would occasionally mix them up. To be expected I guess. Sometimes would flag them as correct on entry, and as the other twin on exit.
I have a largish beard, and found I got the most false hits. Since it was winter (and winters here are cold) a lot of people with touques or hoods. It seemed like basically anyone with glasses and larger blonde/brown beard was registered as me. One of my bosses was frequently missed all together. But I think we solved this through some tweaking of settings.
We're still going through our very long and slow internal process for ordering, but it seems like the company is going ahead with purchase for a bulk of our retail stores.
Interestingly, with my beard, I generally have the lowest confidence of anyone in our office. Other people have facial hair but not with the length I do. I usually top out at about 0.70-0.72 confidence. Other people with shorter hair and beard often are at 0.80 and above, and I've seen some of them at 0.90+.
It was the same for me. I generally always had the lowest confidence. And the false hits were always just barely over the threshold. I think at one point we had gone up to 0.57, and we'd get a false hit for me at like 0.58, or even 0.575
Great test, guys! Curious if in your testing, you would support AnyVision's claim on their website that it works on "any angle and in any light condition" and that it's "plug and play"?
Also, curious to hear IPVM's thoughts about the GUI looking entirely different from what's advertised on their website.
Lastly, how do you think their recent attention from Forbes will impact the involvement of Microsoft?
I wouldn't say it works in any angle and in any light condition. Low light and WDR conditions effectively required subjects to be at a shallow angle in order to recognize properly. Using harsher downtilt led to a lot of misses in those conditions. That being said, it is capable of working at harsher angles and in lower light than others we have seen/tested.
My personal guess is that the Forbest article is unlikely to impact Microsoft's involvement at all, but I'll leave that to someone more knowledgeable in that investment.
Did you guys analyze how any of the hardware performed with different cameras, resolutions, and FPS setting in the given scenarios, day/night, indoor/outdoor scenes and WDR challenged door views or instant lighting changes?
Note: I have not got my hands on Anyvision as of yet. Do they pull RTSP streams for live alerts? such as facial recognition or gun detection?
It was RTSP stream for us. We were told the higher the resolution and frame rate the more taxing it would be on the system. Our test was with a laptop provided by them. Their recommended specs were pretty high.
When you say "any of the hardware" do you mean server hardware? Are you looking for CPU load?
If yes, I did not see any real change in performance of the server hardware based on changes in conditions. We tested with different framerates but didn't see much difference in stream handling, though we did not look at it in depth.
Our server was provided by Anyvision as we wanted to let them spec the hardware so its configuration was not a point of discussion. The GPU in our server is an NVidia 1080 Ti and according to Anyvision is spec'd to handle 5-6 streams, which they defined in our discussions as 1080p or 3MP.
Integration is via RTSP. You can add cameras one at a time or via CSV, but you must manually enter the RTSP stream and credentials.
We've added that to the report, thanks for asking!
Great report Ethan and very timely! I have been waiting for this to compare against Panasonic FacePro - tagging of irrelevant non-facial objects and all. The results are not too dissimilar. However, FacePro has the benefit of not having a bad GUI... basic, but not intolerable. VI helps flesh out FacePros GUI and features. On the down side it only works with VI and Genetec and requires a very limited set of Panasonic cameras. However, the cost is massively different and processing capabilities are greater. Cost per camera is around 1/4 the price of Anyvision. FacePro does have a much more limited set of features compared to Anyvisions advertisements.
This is not an ad for FacePro as all facial recognition deserves a skeptical eye... perhaps this is true for all analytics in general at this point. Anyvision makes some bold claims which I was interested to see tested by a respected third party.
Just a few points of interest based on the testing we have done with AnyVision:
... in our tests Anyvision was unable to detect faces when subjects wore masks over the lower half of their face, regardless of other conditions or settings.
We found that having the nose visible seemed to be key. With the nose covered, no detections. But, we could cover the mouth and chin and it would work, or cover the eyes and forehead and it would work.
...our test server (specified and supplied by Anyvision) would not export clips when GDPR Mode was enabled.
We have not experienced this issue. The system operates slower when GDPR mode is enabled (which can be done on the fly), but it has worked reliably for us.
We found missed recognitions were reduced if we added better quality reference images for subjects. Using clear, well lit photos taken with phones or from social media as reference images produced better results in more difficult circumstances than selecting an image from the UI and creating a subject based around that. Subjects can have multiple reference images, so you could start with a live face capture in the system and add better images if they become available. (Though another annoyance of the UI is that when there are several reference images for a single subject, there appears to be no way to see the ones that don't fit on the screen.)
I think I remember the Anyvision engineer saying something about good to have a nice high quality capture, as you said like from Social media or taken with a camera what ever, and then at least one capture from the on premise camera.
In our test, we had one employee who we put in the system via a picture from a cell phone, and then also added a capture from the entrance camera and he always had the higher capture rate. Even with head tilted, looking at phone, wearing a hat, so on and so forth.
I do agree with the UI part, no way to delete older reference photos either. They all just kind of build up. I hope they change that. When we did our test the only way to remove a reference photo was to delete the subject and re-create.
Probably know about ongoing FR testing at NIST, i.e. Verification and Identification (the former includes AnyVision). I wonder when the open source for this is as good as anything, for example this repo is pretty close to serviceable.
when the open source for this is as good as anything
I don't know but even if the open source was as good, a lot of the premium that people pay is for a turnkey solution. Is there an open source was to simply record video as good as Genetec and Milestone? Maybe but there is a lot of value of making it easy to use, integrated with existing software, adding on functionalities, etc.
I agree John, what I was pointing toward is a leveling of FR technology, approaching a "retail" state, it was an FR/analytics review, I was not talking about video surveillance. It was a counterpoint to the oft stated claim of better FR/analytics.
To your point it impacts product differentiation in a good way. We (when I was building FR in a previous life) used to compete on platforms (architecture, image processing accelerators) and algorithms ( better models and correlation), this is tending towards commodity . Which makes sense, as it should come down to usability, privacy, security and interoperability for the use and service provided not the tech.
Along these lines, here is an excerpt from recent IPVM analysis of an example where open-source shifts the focus of development.
Using Customized Open Source Neural Networks
Scylla said they did not develop their own neural networks, and instead use open-source solutions (e.g. YOLOv3, Faster R-CNN). They said their analytics are able to detect a weapon within 400ms of being seen on camera, at ~40'/12m and ~10PPF. They noted their behavior/violence detection is looking for behavior patterns but is not using pose detection, claiming that the GPU resources are significantly higher for pose.
example where open-source shifts the focus of development
We have spoken to a number of AI startups using open source models. The question is how good adapting / modifying open source models can be vs what proprietary methods. For example, the researchers that developed YOLO, a very popular open sourced AI object detector have started a company, XNOR, that is licensing proprietary software, which they claim is far more efficient. We are working on a profile about them as well.
Yes, completely agree, but others can adapt the tech as well. We use quite a few open source tools with Eidola and our value is what we add in terms of usability, reporting, and increased functionality and importantly I'd like to think our experience. I think an interesting comparison is how practitioners leverage this technology vs. researchers. Nothing like applying open source to do something to advance the understanding of the merit of the code. Will the difference be better AI or will it be better application, system engineering, support and the like? Interesting topic and thanks for engaging.
Yes, completely agree, but others can adapt the tech as well. We use quite a few open source tools with Eidola and our value is what we add in terms of usability, reporting, and increased functionality and importantly I'd like to think our experience. I think an interesting comparison is how practitioners leverage this technology vs. researchers. Nothing like applying open source to do something to advance the understanding of the merit of the code. Will the difference be better AI or will it be better application, system engineering, support and the like? Interesting topic and thanks for engaging.
We can run 30 video streams using commercial GPUs that some of our customers have procured, and it depends on the different types of analytics you would like to run on each of the streams. Your testing was conducted on a an "off the shelf" computer (that we offer for POCs and demos) that was able to run 6 streams, much more is possible on a rack-mounted server.
There is information on what is supported and how it's set up in this Anyvision/BVMS setup guide (PDF link), but little in the way of screenshots if you're trying to just get oriented to what it looks like in BVMS.
Great test and validates the same initial findings that we had when we tested AV last year. It's my understanding that the new GUI and feature set is supposed to released soon.
We have a spin on it's deployment and use.... Instead of using AV as just a face rec system, we are deploying it as an "frictionless access" system as an overlay to standard access control systems. AV has several integrations with partner access control systems (Avigilon, Genetec, Software House, etc..) that will pass positive face "hits" to the access control system to allow/deny access. After a little tweaking and fine tuning, the systems have become very reliable. We have deployed AV on doors, turnstiles, and parking gates and the customers absolutely love it!
Are you using AnyVision's Better Tomorrow offering for the "friction-less access" application you mentioned or AnyVision's Abraxas offering? If you can share, I would like to know what VMS you are using this with.
USD$2500 per license I would have expected much better test results.
Agree
Disagree
Informative
Unhelpful
Funny
Create New Topic
Subscribe to IPVM Research to read the full report.
Why do I need to subscribe?
The IPVM Research Service includes products tests and shootouts plus competitive and financial analysis, helping decision-makers better evaluate purchasing, partnering, developing, and/or competing against companies in physical security.
Comments (33)
Undisclosed Integrator #1
We tested AnyVision back this past winter. My bosses were quite impressed with it, as was I but I had no previous experience with Facial Recognition.
We only had about five employees on the watchlist, and then a series of known actual subjects (20, I think?). Tried a set of identical male twins. It seemed fairly accurate, though would occasionally mix them up. To be expected I guess. Sometimes would flag them as correct on entry, and as the other twin on exit.
I have a largish beard, and found I got the most false hits. Since it was winter (and winters here are cold) a lot of people with touques or hoods. It seemed like basically anyone with glasses and larger blonde/brown beard was registered as me. One of my bosses was frequently missed all together. But I think we solved this through some tweaking of settings.
We're still going through our very long and slow internal process for ordering, but it seems like the company is going ahead with purchase for a bulk of our retail stores.
Nice test!
Create New Topic
Undisclosed Manufacturer #2
Great test, guys! Curious if in your testing, you would support AnyVision's claim on their website that it works on "any angle and in any light condition" and that it's "plug and play"?
Also, curious to hear IPVM's thoughts about the GUI looking entirely different from what's advertised on their website.
Lastly, how do you think their recent attention from Forbes will impact the involvement of Microsoft?
Microsoft Slammed For Investment In Israeli Facial Recognition ‘Spying On Palestinians’
Create New Topic
Undisclosed End User #3
Great test! We should be receiving AnyVision in the next few weeks, and this will definitely help in my deployment.
Create New Topic
Undisclosed Manufacturer #5
From NPR today
Israel Uses Controversial Technology To Screen Palestinians In The West Bank
Israel Uses Controversial Technology To Screen Palestinians In The West Bank : NPR
Create New Topic
Undisclosed #6
Hi Ethan, nice report.
Did you guys analyze how any of the hardware performed with different cameras, resolutions, and FPS setting in the given scenarios, day/night, indoor/outdoor scenes and WDR challenged door views or instant lighting changes?
Note: I have not got my hands on Anyvision as of yet. Do they pull RTSP streams for live alerts? such as facial recognition or gun detection?
Thanks for the feedback. /thumbsup!
Create New Topic
Undisclosed Integrator #7
Great report Ethan and very timely! I have been waiting for this to compare against Panasonic FacePro - tagging of irrelevant non-facial objects and all. The results are not too dissimilar. However, FacePro has the benefit of not having a bad GUI... basic, but not intolerable. VI helps flesh out FacePros GUI and features. On the down side it only works with VI and Genetec and requires a very limited set of Panasonic cameras. However, the cost is massively different and processing capabilities are greater. Cost per camera is around 1/4 the price of Anyvision. FacePro does have a much more limited set of features compared to Anyvisions advertisements.
This is not an ad for FacePro as all facial recognition deserves a skeptical eye... perhaps this is true for all analytics in general at this point. Anyvision makes some bold claims which I was interested to see tested by a respected third party.
Create New Topic
Dan Droker
IPVMU Certified | 08/27/19 07:31pm
Just a few points of interest based on the testing we have done with AnyVision:
... in our tests Anyvision was unable to detect faces when subjects wore masks over the lower half of their face, regardless of other conditions or settings.
We found that having the nose visible seemed to be key. With the nose covered, no detections. But, we could cover the mouth and chin and it would work, or cover the eyes and forehead and it would work.
...our test server (specified and supplied by Anyvision) would not export clips when GDPR Mode was enabled.
We have not experienced this issue. The system operates slower when GDPR mode is enabled (which can be done on the fly), but it has worked reliably for us.
We found missed recognitions were reduced if we added better quality reference images for subjects. Using clear, well lit photos taken with phones or from social media as reference images produced better results in more difficult circumstances than selecting an image from the UI and creating a subject based around that. Subjects can have multiple reference images, so you could start with a live face capture in the system and add better images if they become available. (Though another annoyance of the UI is that when there are several reference images for a single subject, there appears to be no way to see the ones that don't fit on the screen.)
Create New Topic
Salvatore D'Agostino
Probably know about ongoing FR testing at NIST, i.e. Verification and Identification (the former includes AnyVision). I wonder when the open source for this is as good as anything, for example this repo is pretty close to serviceable.
Create New Topic
Salvatore D'Agostino
Yes, completely agree, but others can adapt the tech as well. We use quite a few open source tools with Eidola and our value is what we add in terms of usability, reporting, and increased functionality and importantly I'd like to think our experience. I think an interesting comparison is how practitioners leverage this technology vs. researchers. Nothing like applying open source to do something to advance the understanding of the merit of the code. Will the difference be better AI or will it be better application, system engineering, support and the like? Interesting topic and thanks for engaging.
Create New Topic
Salvatore D'Agostino
Yes, completely agree, but others can adapt the tech as well. We use quite a few open source tools with Eidola and our value is what we add in terms of usability, reporting, and increased functionality and importantly I'd like to think our experience. I think an interesting comparison is how practitioners leverage this technology vs. researchers. Nothing like applying open source to do something to advance the understanding of the merit of the code. Will the difference be better AI or will it be better application, system engineering, support and the like? Interesting topic and thanks for engaging.
Create New Topic
Undisclosed Manufacturer #9
AnyVision is posting that they can work with 30 streams per GPU - this is far away from your test results with six streams - can you comment this?
Is there something known about there integration with BVMS - they are a main investor from AnyVision.
Create New Topic
John King
Great test and validates the same initial findings that we had when we tested AV last year. It's my understanding that the new GUI and feature set is supposed to released soon.
We have a spin on it's deployment and use.... Instead of using AV as just a face rec system, we are deploying it as an "frictionless access" system as an overlay to standard access control systems. AV has several integrations with partner access control systems (Avigilon, Genetec, Software House, etc..) that will pass positive face "hits" to the access control system to allow/deny access. After a little tweaking and fine tuning, the systems have become very reliable. We have deployed AV on doors, turnstiles, and parking gates and the customers absolutely love it!
Create New Topic
Undisclosed Distributor #10
USD$2500 per license I would have expected much better test results.
Create New Topic