AI Video Tester Released

By: IPVM Team, Published on Apr 02, 2019

IPVM has released the world's first AI video tester that lets you see how various AI models (including from Amazon, Google, Microsoft and YOLO) work on your own video.

Why?

While there is lots of hype about 'AI', 'Deep Learning', Neural Networks, CNNs, etc., it is hard to know how well they work. Worse, it is difficult to tell how they will work on your own video rather than marketing demos or generic photos.

How It Works

You can try the AI video tester here by choosing from one of our sample videos or (members can) upload their own video. Below is a sample video from a moderately challenging video surveillance scene analyzed by Amazon Rekognition. Notice it does well generally but periodically thinks it sees a bathtub:

And our Tester maps out where and when each object is seen plus lets you scan the timeline below to see frame by frame which objects are detected, as shown in the gif below:

Use Cases For It

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

We do not expect everyone to make use of this but here are the main use cases we see:

Research

If you, like us, are doing research into computer vision, this tool is a unique means to quickly and easily gauge the performance of various models. That was the first reason we built it, as we build up our computer vision testing, we wanted a way to do faster and better testing. You will see us using this in upcoming reports.

Education

Most video surveillance professionals know little about AI beyond buzz words (e.g., 'neural nets' and 'layers', etc.) but they have little idea how well they actually perform. This tool makes it easy, instead of spending days and having to know how to set up each of these models, you simply add whatever videos you want and let the tester run it for you.

For example, many of our beta testers were surprised about the results they saw, including how poorly many models worked in many challenging surveillance conditions.

We have an upcoming AI Video Analytics course and this tool will be a core component of the exercises and training.

Comparison

For those looking to use these models in production, this will help directly in making product comparisons. Of course, there is a major limitation, the tester does not include any surveillance manufacturers products yet.

Future Improvements

For now, we are releasing the foundations of the Tester. 

The most obvious improvement is more models / systems, from OCR and LPR offerings to various manufacturer's AI systems. While we will add more, we will not ever be able to make it all inclusive since many video surveillance system either do the analytics inside the camera or do not provide sufficient APIs.

Another improvement we are exploring is to make a simple video analysis tool for exported video, i.e., using one or more of these models to help integrators find people or vehicles from long recorded video clips.

We can also add facial recognition to this by using our Tester and adding on a face database component so members can experiment with different systems on their own video before deploying on site.

And, of course, we are definitely open to suggestions from members for improvements.

Try It Out - Give Us Feedback

Try it out, let us know what you think, questions you have and improvements you want.

Comments (15)

Only IPVM Members may comment. Login or Join.

Awesome new IPVM tool guys...!

Really, really, really could have used a tool like this in 2005, for managing video analytics expectations.

Good luck! 

I see the bathtub too. What an awesome toool

Tried two uploaded videos: this was a real crime where a person opens a gate, and the 2nd was him stealing a gator tractor. With the 4 models for object detection: the first two models had a lot of hits off light reflections including mirror and airplane. Models 3 and 4 did not detect anything. 

How can this tool demonstrate how these models can learn?

Hi Robert, these models do all their learning up-front; they don't learn anything new as new videos are uploaded. This is a process called "supervised learning". The model developer starts with a big dataset of videos and known objects within them, trains the model on these videos, and then packages that model up / wraps it in a bow. Now it'll work the same for each run of the same video every time, it doesn't learn more going forward. 

Some vendors tout online learning; learning from new videos as they're uploaded. This might be by way of "unsupervised learning," though I'm not familiar with their approach (something for my radar).

+10,000 for such an educational tool.  Perfect for showing "AI" capabilities and cutting through the hype.

Great idea! We'll try it for sure. According to our experience such a general networks were trained on nontypical for CCTV scenes and work worse in comparison with specially trained ones.

Hi Tyler,

We have been working with the Yolo model and have been feeding it 1000s of clips. We have  3 metrics for evaluating the AI: Accuracy, Efficiency, and Fatal Error. We feed events transmitted from cameras with analytics into the AI model and then compare its classification with how operators classify it in Immix. There are 4 outcomes: #a, #b, #c, and #d. See below. Would like to see Accuracy >95%, Efficiency at 30% and Fatal Error AI below 5%. So far it is not ready for real time.

Thoughts? 

This looks great, and hopefully it can help manage the, often unrealistic, expectations on what computer-vision is capable of (especially when the input is from CCTV cameras with heavy compression and bad light). I tried a few of the IPVM videos, but I couldn't get the graph to show up (I am on Firefox).

If I am not mistaken, the algorithms used are all "still frame" algorithms that treat each frame as if it was completely unrelated to the previous. AFAIK, YOLO doesn't "remember" that it found a person at x,y in the previous frame. It's just so fast that it is possible to do the classification ROI boxes on 30 fps video (maybe they improved this in v3). There certainly are algos that are designed to use knowledge from the previous frame in the next, but I can't remember which.. maybe I'll dig it out one day.

There's a brief video on the NVIDIA deep-stream running on the $99 Jetson nano board. 8 x 1080p @ 30 fps pretty robust object tracking. It's impressive stuff, and perhaps something for IPVM to look into.

https://www.youtube.com/watch?v=Y43W04sMK7I

AFAIK, YOLO doesn't "remember" that it found a person at x,y in the previous frame

You are right! Object tracking based on objects detection from YOLO-style networks is a different non-trivial task.

 

There's a brief video on the NVIDIA deep-stream running on the $99 Jetson nano board. 8 x 1080p @ 30 fps pretty robust object tracking.

There are no identifiers of each object, just bounding boxes. So I think it's not a tracker, but a frame independent object detection. One may count all the people at each moment in time, but it's not possible to trace the path of each person.

This tool is current discontinued? When is it expected to reactivate?

Yes, we discontinued it. We do not have any plans to reactivate it for now. It was not used much and it had high ongoing costs. We may reimplement it in the future but no plans currently. My apologies.

Thanks John. Are there any alternatives to gauge the performance of various models?

I don't know. Obviously people can do it themselves one by one but I don't know of any online services that let you try / compare.

Related Reports

AI/Smart Camera Tutorial on Feb 20, 2020
Cameras with video analytics, sometimes called 'Smart' camera or 'AI'...
Video Analytics 101 on Mar 16, 2020
This guide teaches the fundamentals of video surveillance...
VMS 101 on Mar 03, 2020
This guide teaches the fundamentals about video management...
VSaaS 101 on Mar 25, 2020
Video Surveillance as a Service (VSaaS) is the common industry term for cloud...
Video Surveillance Business 101 on Mar 30, 2020
This report explains the fundamental elements of the video surveillance...
NetApp Video Surveillance Profile on Mar 09, 2020
NetApp is increasing its efforts in video surveillance and told IPVM...
Video Surveillance Cameras 101 on Feb 25, 2020
Cameras come in many shapes, sizes and specifications. This 101 examines the...
Video Surveillance Architecture 101 on Feb 18, 2020
Video surveillance can be designed and deployed in a number of ways. This 101...
Video Surveillance Trends 101 on Apr 01, 2020
This report examines major industry factors and how they could impact video...
Directory of 95 Video Surveillance Startups on May 20, 2020
This directory provides a list of video surveillance startups to help you see...
Sony Launches AI Camera Sensors on May 18, 2020
Weeks after exiting the branded video surveillance business, Sony is making a...
AndroVideo Presents Edge AI Face Recognition Cameras on Jun 26, 2020
AndroVideo presented its AI at the edge face recognition cameras at the May...
Last Chance - Spring 2020 IP Networking Course - Register Now on May 06, 2020
This is the last chance to register for the only networking course designed...
Surveillance Storage 101 on Mar 23, 2020
This guide teaches the fundamentals of video surveillance...
Video Surveillance History on May 06, 2020
The video surveillance market has changed significantly since 2000, going...

Recent Reports

Dangerous Hikvision Fever Camera Showcased by Chilean City on Aug 07, 2020
Deploying a fever camera outdoors, in the rain, with no black body, is...
"Grand Slam" For Pelco's PE Firm, A Risk For Motorola on Aug 07, 2020
The word "Pelco" and "grand slam" have not been said together for many years....
FLIR Stock Falls, Admits 'Decelerating' Demand For Temperature Screening on Aug 07, 2020
Is the boom going to bust for temperature screening? FLIR disappointed...
VSaaS Will Hurt Integrators on Aug 06, 2020
VSaaS will hurt integrators, there is no question about that. How much...
Dogs For Coronavirus Screening Examined on Aug 06, 2020
While thermal temperature screening is the surveillance industry's most...
ADT Slides Back, Disappointing Results, Poor Commercial Performance on Aug 06, 2020
While ADT had an incredible start to the week, driven by the Google...
AHJ / Authority Having Jurisdiction Tutorial on Aug 06, 2020
One of the most powerful yet often underappreciated characters in all of the...
SIA Coaches Sellers on NDAA 889B Blacklist Workarounds on Aug 05, 2020
Last month SIA demanded that NDAA 899B "must be delayed". Now that they have...
ADI Returns To Growth, Back To 'Pre-COVID Levels' on Aug 05, 2020
While ADI was hit hard in April, with revenue declining 21%, the company's...
Exposing Fever Tablet Suppliers and 40+ Relabelers on Aug 05, 2020
IPVM has found 40+ USA and EU companies relabeling fever tablets designed,...
Directory of 201 "Fever" Camera Suppliers on Aug 04, 2020
This directory provides a list of "Fever" scanning thermal camera providers...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Dahua Loses Australian Medical Device Approval on Aug 04, 2020
Dahua has cancelled its medical device registration after "discussions" with...
Google Invests in ADT, ADT Stock Soars on Aug 03, 2020
Google has announced a $450 million investment in the Florida-based security...