There have been at least two attempts to create a "Consumer Reports" type of evaluation service for security and surveillance products in the past. One was started by a consultant like myself, the other by a self-proclaimed "security thought leader". I won't identify their names.
The problem with one was that it relied on manufacturers to support its business model. Each manufacturer would pay to have their product rated and provide the consultant with free equipment to test. In my mind, this service was unfair as it only provided ratings on those manufacturers who "paid to play". I'm not sure that the service ever developed a critical mass and soon fell by the wayside.
The other service had similar conflict of interest problems, as the the person running it would obtain his information primarily through schmoozing with manufacturers and attending industry events rather than conducting independent tests. This guy also had consulting gigs on the side with manufacturers that I felt made his advice anything but objective.
Another person in the industry approached me about 15 years ago and asked about starting a similar service. My advice to him at the time was that while such a service would be valuable, very few people would be willing to pay what it would take to make the service self-sustaining, particularly if the equipment had to be paid for rather than provided by the manufacturer. I did not believe that enough people would be willing to pay a large enough subscription fee to keep the service afloat. I am delighted that John has proven me wrong about this.
As valuable as the IPVM training and product testing is, I feel that the discussion forum is of equal or greater value. Unlike forums that anyone can join, IPVM members have paid a fee to in order to participate, which I feel does a good job of screening out the flakes. When you look at the depth and breadth of the IPVM membership, you have a good representation of the key players in our industry.