Makes sense.
What I was suggesting was instead of IPVM performing the tests directly, as this is cost and time prohibitive, that IPVM determine the most typical use cases and create or crowd-source a series of progressively more difficult video footage along that same scenario.
For example, cross line detection along a fence.
Level 1 : Empty Scene, great camera positioning, object larger and closer to the camera and thus easier for VA to detect and filter out nuisance alarms.
Level 2 : Same as before but object farther away and thus smaller (i.e. represented by smaller pixel density)
Level 3: Scenario that introduces shadows, or a scene where the clouds are shifting and the sun shines through causing a massive change throughout the entire scene.
Level 4, 5, 6, 7, 8. progressively get more and more complex.
This way over the next 8 years we can see if any VA manufacturers want to take a go at the challenge and we can then discuss their results.
You could make it even more challenging if a VA mfg can produce a result across all levels using the same configuration and without significant tuning of settings per each challenge.
This way the burden of the initial testing would be on the various manufacturers and if after they have succeeded at a number of scenarios and difficulty levels, only then would IPVM setup an independent testing to validate their submissions.
NOTICE: This comment was moved from an existing discussion: Manufacturer President: "Customer Is Now Very Angry"