Subscriber Discussion

Witness / Acceptance Testing Of Video Surveillance Systems?

ML
Michael Leslie
Dec 29, 2016

In analogue days we used to witness test installed systems using a rotakin test target to measure and confirm that the required resolution and FOV specifically the image height for detection, recognition and identification have been met.

What are the best methods or tools that are recommended today for witness testing an installed IP camera systems to measure the resolution and the design intent have been met, including the dynamic range and performance e.g. low light etc have been accomplished?

JH
John Honovich
Dec 29, 2016
IPVM

Michael, that's a very good question, thanks for sharing!

My gut feel is that most users do not have a formal / quantitative process of verifying / validating performance. Most typical is likely a subjective walk through where a tech or employee walks from camera to camera and a manager views it on the system.

Many do specify PPF / PPM up front (i.e., minimum 50ppf, 150ppm, etc.) and that can help somewhat but, as you point out, for aspects like dynamic range and low light performance, PPF / PPM does not help.

Let's see what others do.

(3)
Avatar
Robert Baxter
Dec 30, 2016

We walk test the site at night to confirm the analytics are picking us up when installed. Been doing that for 18 months, huge benefit.

We are in the process of setting up all our permanent sites for annual inspections and so we are comparing the images of night shots against the installed views as reference. I believe this will give indication of failing IR, cleaning and Rain-X application of both cameras and IR. It sounds simple but executing it can get complicated quickly.

Would value any other suggestions.

(1)
(3)
ML
Michael Leslie
Dec 30, 2016

Robert, I believe that is a good practice you are adopting for ongoing quality monitoring, but I am interested in the initial set up and trying to measure that the performance specified for the system has been accomplished and the design intent has been met prior to hand over to the end user.

I would be most interested in your recommended processes that you follow to ensure the initial performance of the video surveillance system has been met? How do you measure/analyse/confirm the design criterion has been met?

Avatar
Robert Baxter
Dec 30, 2016

We start with a minimum pixel density for reliable analytics detection (based on avigilon guidelines). We are not meeting any 3 rd party specification. Real conditions like objects, vehicles can obscure the available pixels. Our criteria is getting arrests and not having misses. The customer use of the video maybe important initially but often they lose interest in looking at their cameras.

(1)
Avatar
Ross Vander Klok
Dec 30, 2016
IPVMU Certified

I agree with John's take.  As an end user our process is 100% subjective.  "OK, how does it look now?  Good?  OK, great I will button it up." We do a walk through on all three shifts to make sure we don't need to tweak anything due to lighting or activity differences and we also check recorded footage for the first few days to make sure motion is triggering recording as it should.  Other than that we don't do much.

I would be very interested in ideas on how to do this in a more rigorous, scientific fashion though.   Great question Michael!

(1)
UI
Undisclosed Integrator #1
Dec 31, 2016

Robert, I've been looking at different video analytics and have played with Agent Vi. I've noticed that Agent VI has been been giving an unusual amount of false detections in the rain/glare -- Have you noticed the same performance with Avigilon's video analytics? 

If you do prefer Avigilon, you're restricted to only using Avigilon cameras to detect analytics?

(1)
Avatar
Robert Baxter
Dec 31, 2016

UI#1

Not used Agent 4. Not sure what an unusual amount means anymore. 50% of our 130,000 events per month is due to Reflection, Weather, and Wildlife in that order. It can rain a lot here...

You can use other cameras with the Rialto units.

(1)
UI
Undisclosed Integrator #1
Dec 31, 2016

Just out of curiosity Robert, how many cameras are generating those 130k alarms?

Avatar
Robert Baxter
Jan 01, 2017

Just over a 1,000.

U
Undisclosed #2
Jan 01, 2017
IPVMU Certified

Robert, does the Avigilon Analytics allow you to switch between multiple settings based on time-of-day or other events?

For instance, if rain was detected by one analytic, it could lower the sensitivity of another analytic.

Avatar
Robert Baxter
Jan 01, 2017

Its a good idea to detect rain and do something different. I am not aware of that capability. It might not make a difference...

We have also tracked the number against the mm of rain in a week - there is a correlation but if it isn't one thing its another.

 

   

(1)
U
Undisclosed #2
Jan 01, 2017
IPVMU Certified

Do July and Aug benefit because of longer days?

Avatar
Robert Baxter
Jan 01, 2017

Not sure I get your question.

U
Undisclosed #2
Jan 01, 2017
IPVMU Certified

Offhand it looks like Jul and Aug have the fewest false alarms.  I was wondering if that could be attributed to more daylight in those months (as well as the reduced precipitation).  

Assuming you are in the Northern Hemisphere, of course.

Avatar
Robert Baxter
Jan 01, 2017

Longer nights more reflections

Better weather more people walking about

Poorer weather more reflections and weather false positives

I suppose.

MM
Michael Miller
Jan 01, 2017

Robert are you using Rialto units with 3rd party cameras or Avigilon H4 cameras with analytics?

Avatar
Robert Baxter
Jan 02, 2017

Yes and Yes. Mostly (analog and IP Rialtos) and a few H4A, also VideoIQ 1 channel encoders, and Videoiq HD icvr.

UI
Undisclosed Integrator #1
Jan 02, 2017

Robert,

When you use these analytics, I'm assuming you're putting them on their own dedicated VMS at their local site and then connecting that VMS back to whatever CMS software you're using at the monitoring center?

Is there a specific VMS that you generally install for clients, or whatever the client prefers?

This has been so informative, thanks for sharing!

Avatar
Robert Baxter
Jan 02, 2017

We approach it differently and program the appliances to communicate with the CMS. We would create a separate stream to their VMS from the cameras.

ML
Michael Leslie
Jan 02, 2017

Ross, Thank you for your views as an end user, which I agree provides a good procedural methodology, especially over the staff shifts to allow for different lighting conditions. Do you take reference images over the three shifts for ongoing analysis to monitor system performance and to monitor seasonal changes?

Avatar
Ross Vander Klok
Jan 04, 2017
IPVMU Certified

We do exactly that Michael.  We have a simple Excel spreadsheet that not only allows the Officers to quickly get to the cameras they need but also assists with the monthly audits.  We audit camera view, bandwidth being used and verify that it is recording properly.  Some audits are weekly and some are monthly.  Attached is a screenshot of what is found for the correct orientation and overall quality of a camera.

I should say the spreadsheet used to be simple.  Now that we have roughly 175 cameras around the globe the sheet is getting a little unruly...  Plus we still have the issue with Officers not knowing if the camera is still on the old analog/DVR system or on the spreadsheet.  All of that is off topic stream of thought as I type!

Day time capture

(1)
ML
Michael Leslie
Jan 05, 2017

Ross, I believe that the audit processes you have adopted to monitor the ongoing performance, will prolong the optimum quality of your video surveillance system. The original reference video images are always useful for comparison with current conditions, particularly following any routine maintenance and repairs or when replacements are required.

DW
Donna Whiteside
Jul 21, 2020

As a camera test; if a sign were created, to test a cameras's resolution at 50', how large would the font need to be to simulate the required pixels/foot? That seems to be a measurement that would be repeatable. My camera design criteria, is 60ppf at 50' with the camera zoom all the way open, 4.3mm.

This is an Axis Q3517-PLVE, 5MP, 1/1.8 chip, with the zoom full width at 50' the calculation would give 26 ppf. My contention is the design firm didn't provide enough cameras to satisfy the design criteria, now I need to provide it with a test that isn't subjective.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions