How Should We Test / Shootout Camera Management / Bulk Configuration Tools?

To address the first obvious question:

We are testing Avigilon, Axis, Bosch, Dahua, Hanwha, and Hikvision tools.

Hopefully this will be a composite list from multiple people who have purchased VMS discovering performance fell short their expectations. 

1. Easy of setup, camera enrollment.

2. Once cameras are enrolled, are the features you chose the cameras for still available in the VMS?

    Example 1: the camera allows you to rotate the camera view to view corridor better but the function is ignored by the VMS. 

   Example 2: 360 degree camera has function to provide a quad view, but the VMS will only present the fish eye view.

3. How many different  cameras can be used.

4. How often are camera SDK updated.

5. Can the VMS multicast?

6. Can video be quickly dispatched to police officer's phones.

Maybe something else will come to me later.

Exporting discovery data (spreadsheet/csv).

Perhaps identify which tools can easily be downloaded from a website versus other tools that require authentication, login or certification. If any attacker can just grab the tool and start learning(hacking), pulling down firmware looking at how the system internals work he has an easy shoe in to reverse the firmware, decompile and reassemble the tool and camera firmware; etc. Ever take some hikua firmware and look at the logical versus the physical size of the file, then compare it with another (same version) firmware? You might find firmware versions such as H2.23.1111example.bin changes depending on where you get the firmware from. A US site or China site, it's always fun when you find two firmware with the same dates, revision, everything but for some reason they are different in size(the logical size between the two differ as well as the physical size).

It's always helpful if the tool can span subnets, not just discover cameras on the same net as the PC running the tool.  I think this is typically a function of whether the tool uses a broadcast or multicast packet for discovery.  When bench-config'ing, not so much a big deal.  But if you're on a client's network trying to run a discovery, it's great when you don't have to visit multiple IDFs.  I usually also have a generic IP discovery tool handy - can find all IP devices, sort by MAC, and determine what cams are out there.