BETA - AI Tester - Check It Out - Give Us Feedback

JH
John Honovich
Feb 13, 2019
IPVM

We have a new / upcoming software tool, called the 'AI Tester' that allows any IPVM member to see how different AI systems (from Amazon to YOLO, etc.) work on one's own videos. This way you can see, before buying or setting anything up in your own facilities), how these systems work with video from your own environments.

You upload your video (or pick one from our samples), pick a model you want and see how well it works. It then returns the results overlayed on an output video, like so:

We have a number of plans for improvement and we will do a formal announcement soon but wanted to do a limited release with members on the discussion forum before that. What problems or what enhancements would you like to see? How do you see yourself using this?

And thanks to Tyler Renelle who recently joined IPVM as a machine learning engineer who built this!

(2)
(4)
SD
Shannon Davis
Feb 13, 2019
IPVMU Certified

So running a short clip with three people walking within 10' of a camera there is a huge difference among the analytics. So far the Rekognition model is the best at detecting faces in the clip I provided. Some didn't even detect was a face. 

Avatar
Tyler Renelle
Feb 14, 2019

In the near future we'll have a models side-by-side comparison, with indicators where objects were detected in one model but not another, etc.

(3)
JH
John Honovich
Feb 14, 2019
IPVM

And we are going to add in more models / systems. For example, Tyler just added in Amazon Rekognition object detection so, e.g., you can compare / contrast it to YOLOv3.

Any suggestions for others to add let us know? Google Cloud AI? Microsoft Azure? Others?

In particular, ultimately we would like to add in commercial systems. For example, submit a video clip and see how Avigilon or Hikvision etc. does.

JH
John Honovich
Feb 14, 2019
IPVM

Btw, I was just trying Amazon Rekognition object detection and it detected a bathub:

My guess is that it was not trained in black and white images and so it thinks the oval / white shape is a bathtub from a color image. Thoughts?

(1)
(2)
Avatar
Tyler Renelle
Feb 14, 2019

I can decrease MaxLabels (currently detects 123 objects, I could set it to 10 or such) and increase the MinConfidence. See API details. Let me know if you want me to try that.

JH
John Honovich
Feb 14, 2019
IPVM

Tyler, the bathtub generated an 87.15% so that's pretty high confidence. For example, if we raised it to 90% it would then lose the accurate matches of the 2 cars in the background.

I do wonder if maybe we let people set/adjust the confidence level as an advanced configuration setting on the AI Tester but that's secondary.

U
Undisclosed #5
Feb 24, 2019

I think this is clearly the case - the shape and lighting creates at least something similar to what a bathtub might look like maybe.

If it recognized the rectangle as a giraffe then I would just assume it was Dahua object recognition analytics.

(1)
U
Undisclosed #1
Feb 14, 2019

Can you upload URL video? 

https://www.youtube.com/watch?v=YujTLFRJL7E

The results would be amazing. Muhaha.

Avatar
Tyler Renelle
Feb 14, 2019

Not yet, I'll look into adding that. I could just subprocess the CLI tool youtube-dl, easy - but I'll have to check into legal concerns.

(1)
SD
Shannon Davis
Feb 14, 2019
IPVMU Certified

Really fascinating how different the performance and reliability of the different analytics in the different manufacturers. I would sure hate to purchase one of these as a customer then find out what I just spent thousands on doesn't really work that well. Oh wait a minute the demo video they showed me worked flawlessly. Great work IPVM on this!!

(2)
Avatar
Tyler Renelle
Feb 14, 2019

Thanks Shannon!

UI
Undisclosed Integrator #2
Feb 14, 2019

Is that the snow in Hawaii?  :D

(1)
UM
Undisclosed Manufacturer #3
Feb 15, 2019

I love and appreciate what you're doing but hate the generalization of "AI" and "analytics" as a description as some sort of universal gathering of data that is inherently intelligent and useful.

In order for this to be real world meaningful, the detection goal must be defined first and then the comparitive results of tests presented with probability of detection, probability of false positive, etc. Really hard to do scientifically with so many scenarios of camera placement and fields of view .

 "I have camera or system with awesome AI and analytics." 

"For what?"

"Dude crossed a line."

"I thought analytics was to capture faces and notify me about a bad guy entering my place."

"No, AI is for license plate capture so I can know when bad parent enters property."

"Will AI let me know when something bad happens?"

AI and analytics have fairly specific use cases. Very important to analyze, compare and detail any analytic claim against the real world use case.

Just a thought.

(1)
(1)
JH
John Honovich
Feb 15, 2019
IPVM

In order for this to be real world meaningful, the detection goal must be defined first and then the comparitive results of tests presented with probability of detection, probability of false positive, etc. Really hard to do scientifically with so many scenarios of camera placement and fields of view .

That's up to the user uploading video. They can make their own judgments about how well each model or service works. 

hate the generalization of "AI" and "analytics" as a description as some sort of universal gathering of data that is inherently intelligent and useful. [emphasis added]

We are not making such claims here. We allow users to try out different AI models and make their own decisions about how they work.

(2)
UM
Undisclosed Manufacturer #3
Feb 16, 2019

My apologies, I was not implying you are making such claims. Just revisiting/ranting a thought that drives me crazy when the topic of intelligent video comes up.

UI
Undisclosed Integrator #4
Feb 19, 2019

Oh man that Amazon Rekognition loves faces....  Great tool - thank you!

 

(1)
JH
John Honovich
Feb 19, 2019
IPVM

#4, thanks for sharing.

Related update: We now have 8 face detectors in the Tester, with Tyler recently adding in Google Cloud and Microsoft Azure, partial screencap:

(1)
(1)
Avatar
Sean Nelson
Feb 19, 2019
Nelly's Security

cool. How long does it normally take to process? been stuck on this screen for several minutes and different attempts:

Avatar
Tyler Renelle
Feb 19, 2019

Looks like the job-queue got "clogged" somehow, I restarted it and it ran a ton of backed-up. I'll look into this & fix today

U
Undisclosed #1
Feb 25, 2019

Tyler, how deep do you go? Do some manufacturers give you code samples? Perhaps sign a NDA here and there?  Are you just performing out of the box mock ups with local field tests?

I wish there was a...pre-prod deployment service that allows SMEs to access a video analytics deployment AI for the sake of consulting on performance or compliance issues.

An AI SME could consult camera angles, camera views, layout and operability to the human interface along with recommendations that maybe cliché standards per the customer/stakeholder expectations.

As far as I know, no consultants can blog on AI, Video Analytics right now due to the hardware, software infrastructure needed for performance.

IPVM has a calculator, do you see an AI, Analytics feature you can embed into that calculation? Camera based, Cloud Services Based or On site Premised based servers (think about all those Tesla V100 cards).

I look forward to you reviews and insight.

JH
John Honovich
Feb 25, 2019
IPVM

We don't sign NDAs generally.

I am not sure what you are saying. For example:

As far as I know, no consultants can blog on AI, Video Analytics right now due to the hardware, software infrastructure needed for performance.

No consultants can blog on AI? What does that mean?

IPVM has a calculator, do you see an AI, Analytics feature you can embed into that calculation? Camera based, Cloud Services Based or On site Premised based servers (think about all those Tesla V100 cards).

What does Tesla cards have to do with the IPVM calculator?

JH
John Honovich
Mar 01, 2019
IPVM

Update: Tyler has added in TinyYOLOv3 so you can, e.g., now easily compare TinyYOLOv3 to 'full' YOLOv3 to see differences in accuracy and speed.

Also, I was just running a comparison between Google Vision and Amazon Rekognition, Amazon did notably better, catching more people with no false matches. Google Vision, interestingly, momentarily mistook this shadow for a surfboard:

It does roughly have a surfboard shape, though clearly not to a human eye.

(3)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions