A Different Structure To Do A Lot More Camera Tests?

There are thousands of camera models from hundreds of manufacturers. Covering the costs of testing even a small number of them is daunting, especially since most people are not interested in 95% of the cameras on the market. Most camera tests deliver very low reads. On the other hand, the small number of people who are seriously considering using those cameras might find significant value.

This is the challenge I find. It costs far more to test a camera than any other type of report (like 10x - thousands of dollars vs hundreds) but most camera tests deliver very few reads compared to educational, news, trends, or survey posts.

The tests that do the best are shootouts and new technology ones as they have broad appeal. Those we see deliver significant value.


What if we charged a separate fee to access niche camera tests? Shootouts and new technology tests would remain available to all members but individual camera tests would cost extra.

The reality is, even as we continue to grow robustly, spending on niche camera tests, is a bad investment for our members because most do not care for them. On the other hand, a small percentage would find them valuable and this would be a way for us to deliver them.

Btw, under no circumstances will we accept payments from manufacturers to do test. That's not negotiable. IPVM has to be able to cover costs of tests by ourselves.

What do you think?

I would love a deep dive into how/what level integration a specific camera has with multiple VMS platfoms. The bottom line to me (and I would think to most of the integrators and end users) is that this is where the rubber meets the road. I want to know what settings/options/programming has to be done in the camera (each and every camera in a 100+ camera install) vs what can be set within the VMS itself. This obviously doubles the complexity of the tests exponentially (if that's even possible), but would deliver the most value to me, and I could easily talk my company into additional fees for those types of reports.

You could even do it Kickstarter style, have people pledge $X and give it 2 weeks and when a critical mass of people are requesting the report, its funded.

That's probably a good example of a report we could cover costs by doing it separately but as part of normal membership.

Maybe it's Axis cameras integrated with 10 leading VMSes or Sony with 10, etc.

On the other hand, we could probably aggregate this information based on manufacturer specs only (sans testing) and include it for general members.

We could conceiveably create a feature for members to pledge or pre-order niche reports and then trigger them when it hits a certain level. Would others be open to that?

"On the other hand, we could probably aggregate this information based on manufacturer specs only (sans testing) and include it for general members."

My problem with this is similar to the SDK and ONVIF subjects from earlier this week, a lot of VMS manufacturers will list a camera manufacturer or model breakdown of "supported" cameras, but in a lot of cases that can be anything from barebones ONVIF Generic video streaming, motion and analytics flags accepted from the camera, to set color saturation/brightness/mic sensitivity levels - all with varying degrees of hair pulling required to ultimately get the camera working as advertised.

Purely from a customer 'demand' perspective, testing all these low level elements is probably lower priority.

I do think IPVM should soon do a new round of ONVIF testing, because understanding the overall strengths and weaknesses of that 'technology' is core to our member base.

"We could conceiveably create a feature for members to pledge or pre-order niche reports and then trigger them when it hits a certain level. Would others be open to that?"

I think that would be a good system as it would ensure that the interest is there before going to the expense of testing a niche product. If it's a product that I have an interest in or a potential application for, I'd be willing to pay an additional fee. Maybe there could even be a mechanism for members to submit a product suggestion for potential testing.

I'll admit though that my main interest is in the product shootouts or tests of more "mainline" products (which recently are more like mini-shootouts anyway).

This is potentially an idea that could be combined with the "Pay Integrators for Field Reports?" concept - sometimes integrators (or installers, or whoever) out there are able to test newer or "niche" gear in the course of a particular job. Case in point: I'm needing to evaluate the Axis P12 cameras for a current job, coming up with some way to integrate them into particular outdoors structures and determining IF they're even suitable given the specific lighting concerns. I have demo models of both P1204 and P1214-E for this purpose (on loan from our Axis supplier, although the customer has expressed interest in purchasing one outright just to have for future use and/or testing).

The problem I have with integrator tests is that they tend to be informal and have basic errors. For example, settings are not normalized (the FoV of two cameras are different, the tests are done at different times of the day, the minimum exposure is left at defaults). In fairness, for an integrator's purposes, the goal is to determine whether it's good enough for their use, and such tests typically are sufficient for them but would raise fairness and accuracy issues for us. Also, if its niche gear, I am concerned about the resale relationship of the reviewier.

For that reason, I am a lot more comfortable with reviews on general applications than products.

Well naturally, the tests/examples/etc. will be more oriented toward the needs of a specific installation, rather than A/B comparisons. Still, it could be a good way to get some insight on products you may not otherwise have a chance to test.

It's useful to get that feedback from specific people we trust have the skills and expertise to do so but as a general program I think it introduces a lot of risk.