Readers Respond: 134 Votes, 25 Comments Examined

By: John Honovich, Published on Feb 14, 2010

Over 100 people voted and more than 25 left detailed comments to our recent question/discussion. In this post, we review and respond to the comments offered.

134 people responded to our survey on "How Critical Should We Be of Video Surveillance Products?". Here are our results:

Non-manufacturers

For the 104 non-manufacturers responding, such as integrators and end-users, the breakdown was:

  • 68%: Just Right
  • 9%: Too Critical
  • 23%: Not Critical enough

Manufacturers

For the 30 manufacturers responding, the breakdown was:

  • 60%: Just Right
  • 7%: Too Critical
  • 33%: Not Critical Enough

We are glad to see that readers value critical coverage and pleasantly surprised to see the strong numbers of manufacturers who want us to be even more critical (given that their products are the subject of such criticisms).

Reader Comments and Our Response

Below are 25 reader comments and our feedback. We found them to be very interesting and thought-provoking. Thank you!

Reader comment: "Highlight what works, not just what does not. Start accepting sponsored ads (j/k!)"

Our response: At the beginning of our test summaries, we usually lead with 1 or 2 key positives. In our conclusions, we note ideal/best fits. In the middle, a significant portion of the reports is critiques (things that either do not work or work poorly under certain conditions or for users with certain skill sets. We always link to the product's marketing material/datasheets for those who want the 'pitch'. This being said, we will keep this in mind. If there are specific ways to highlight what works, please share.

As for accepting ads, it's great that we do not because it makes it easier for manufacturers to trust us. Most media/analysts are always hitting up manufacturers for money which creates a questionable relationship.

Reader comment: "Be clearer about the methodologies and criteria used.  How does your testing/analysis relate to the real world applications of the products?  For what types of customers and environments? Supporting what use cases?  Etc."

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Our response: We have guidelines but have not shared them broadly enough nor have we refined them as much as we should. See existing guidelines: IP camera guidelines,exposure setting guidelines and VMS evaluation criteria. We are going to revise these guidelines in the next few weeks and provide more opportunity for feedback.

Reader comment: "Deeper analysis of bandwidth & CPU peformance in client PCs, particularly with multiple H.264 streams & scenes with movement and/or marginal lighting conditions"

Our response: We think performance on client PCs is an important point that we have not covered. We think such testing will be especially valuable for use of multiple megapixel cameras. We plan to add this in the next few months.

Every camera tests includes marginal lighting conditions (with new tests including tests at a measured .3 and 1 lux as well as outdoor, street lighting).

Reader comment: "More evaluations of video analytics: think about typical applications like perimiter crossing or loitering in sensitive areas. Ease of set-up & false alarm rate."

Our response: We only have 1 video analytic test (on VideoIQ). This week, we are starting a new test on video analytics and plan to increase this over the next few months. We will likely to do fewer tests of video analytics than VMS or cameras as the market demand is much higher for than the later.

Reader comment: "Less emphasis on the camera's HTML user interface, bearing in mind that the general idea is to integrate most if not all camera-specific parameters into a commercial VMS interface, or in any case only adjust camera parameters during the initial set-up phase."

Our response: We spend significant time of the web interface because many of the high end features of cameras are only available for configuration on the web interface.

Reader comment: "Johns' nirvana seems to be consumer-ready products at dirt cheap prices.  I'm a security system integrator, and if that happens I'm out of a job.The reality is that this stuff is complex, the market is complex, and installation and configuration should be left to professionals.  While I enjoy the product reviews, I think ease of use should be measured with a yardstick, not a whip."

Our response: I think that's a good point. While I don't literally dream of 'consumer-ready products at dirt cheap prices, I believe these are both two key values for users. As such, we focus on them. At the same time, we spent extensive time in both screencasts and reports explaining complexities to help integrators.

Reader comment: "Benchmarking and comparison of multiple products and suppliers is what the industry needs.  I would look forward to see more benchmark comparisons in de VCA video content analysis field.  All too many suppliers make own statements of performance without critical view."

Our response: We think benchmarking is valuable but want to have 

Reader comment: "We need full disclosure of all system features, functions and benefits (and issues) - all measured on the same yardstick."

Our response: See existing guidelines: IP camera guidelines,exposure setting guidelines and VMS evaluation criteria. We agree that disclosure is important. If there are specific features or functions you want disclosure on, please suggest.

Reader comment: "Maybe more cost justification on given products. Cost based on features that address very limited market expectations can often be a deal breaker.  ROI on a single component, not just on the 'system ' seems to be the building block of value to an end user."

Our response: We agree. We did a business case analysis on megapixel cameras last year and have a general examination of Video Surveillance ROI. Those noted, we good definitely do more of those. Any suggestions on specifics would be welcome.

Reader comment: "Megapixel vs HD race update.  More products have been introduced since the laste  article about a year ago.  Thanks for a great site."

Our response: We have a new megapixel comparison report being released next Monday that will provide a major update. Agreed that it is time.

Reader comment: "Include actual clips of the video under the test conditions."

Our response: We do provide actual clips of the video for every camera test. For examples, review our premium service preview that includes downloads from one MP camera.

Reader comment: "1. I'd like to learn more about analog camera, e.g., WD camera, Spoeed dome camera,...2. Could you make a comparison the performance for using differents IC (ASIC, SoC, DSP) to IP camera and to DVRs?"

Our response: We do very little testing on analog cameras (one test on EzWatch's low end kit just to get a baseline of very inexpensive analog). As IP is the growth market, our focus is analyzing those products - both advantages and disadvantages (like the crazy debate on ease of installation).

Reader comment: "Benchmarking and comparison of multiple products and suppliers is what the industry needs.  I would look forward to see more benchmark comparisons in de VCA video content analysis field.  All too many suppliers make own statements of performance without critical view."

-- and related ---

Reader comment: "I am relatively new to the site as a subscriber of two months.  I was attracted to this site because of positive and negative information in the reviews and if IP Video Market stops offering honest criticism to become a clearing house of product commercials then I would not have a need to renew my subscription.  I would like to see the reviews arrive at more of a conclusion such as a rating system (similar to Consumer Reports).  While it is true that a rating system risks angering manufacturers that suffer from poor ratings, if the rating criterion is clear, then they will know what they need to do to improve their ratings.  Perhaps a process that allows manufacturers a way to respond to criticism would add to the fairness and keep companies with poor reviews involved.  If a manufacturer decides that they do not want to participate in reviews because of bad results instead of responding to criticism and correcting problems then that says a considerable amount about their company."

Our response: I do not like rating systems for a number of reasons: (1) ratings obscure differences in value for different applications (a product for a community bank might rate a 9/10 but only a 2/10 for use in a military base), (2) doing ratings presume testers have outstanding comparative testing and understanding. I think that's a high bar to reach that most testers do not reach and simply subjectively guess. (3) Worse, ratings are best for manufacturers because it allows them to use reviews in marketing promotions. Often such ratings have hidden funding from manufacturers (especially and not surprisingly the top rated ones). This being said, I am definitely open to discussing ways of better summarizing or comparing products.

Reader comment: "I would like to see a summary comparison where the results (positive and negative) are tabulated according to the features like consumer reports."

Our response:

Reader comment: "Criticism on products - including 'tricks' that skew claims of fact or specs - is awesome!  Keep everyone honest and make comparisons more fair and just.Criticism on marketing - which are more general - sometimes goes overboard.  Marketing is not always evil or misleading as it is usually positioned.  Some would argue it is necessary to fuel growth, or at a minimum, to survive."

Our response: I have grown somewhat more tolerant of generic marketing claims. What concerns me are marketing claims that significantly over-reach or misrepresent (see our debate on 'riskiest companies'). I think some of these more extreme cases of over-marketing hurt everyone including other manufacturers who either need to respond in kind or lose ground in public opinion. In these cases, we think our critiques can be beneficial for the overall market.

Reader comment: "Ruthlessly expose the marketing claims of manufacturers and the lack of uniform standards in critical performance metrics such as lux ratings for cameras."

Our response: We try to be relentless but not ruthless. We want to be fair to all sides and mainly resist when we believe a manufacturer is hiding performance problems (though this is a minority of cases).

Reader comment: "Would like to see more standardized tests.  Especially of resolution, at , say, 3 different light levels.  Both IP and analog cameras are usually well short of their advertised levels in my experience.Also total file sizes of a standard moving and fixed image would be helpful.  Because for example, H.264, JPEG, etc have many possible implementation subsets and compression efficiency."

Our response: We do our camera tests at .3 and 1 lux as well as an outdoor intersection scene to provide measured references. 

Reader comment: "I believe that you show a bias against IP-based products in favor of analog manufacturers.  The tone and quantity of your critical articles about IP products seems to be overly vehement against."

Our response: A site called "IP Video Market Info" and spend 99% of their coverage on IP is probably not biased against IP video. The IP manufacturers obviously say good things about IP. They also spend significant money buying or writing articles in trade magazines who then also say good things. We see our site as an independent balance that pushes critical debates on IP's flaws. We think this will foster a stronger and more mature market. If we were vehemently against IP, we'd go pull coax cable.

Reader comment: "I am an end user of CCTV products and one of our critical needs is to know and understand all the operational functionality, shortfalls and upsides of a given product/process. And the viewpoint for this is most often from the Control Room operator - what does this new gizmo give him/her; what advantage does it bring into the Control Room and how is his/her job going to be made easier?I look forward to your reviews with keen interest and find them refreshing, useful and thought provoking, so keep it up!"

Our response: Good point. We have done a poor job covering the basics as most of our focus has been on researching newer products and trends. We think your suggestion is valuable and will remember to provide more coverage on the fundamentals.

Reader comment: "We'd like to see tests for Avigilon, Lumenera, Iqeye PRO, Luxriot, Bosch, SeeTec. Also it would be great if Ipvideomarket start to post comparisons between IP cameras with 'for' and 'against' in one table."

Our response: We are speaking with or in progress with most of the vendors on the list. For most vendors, it's simply a matter of time. Luxriot should be released next week.

Reader comment: "more group tests please. Tests like, cameras from different manufacturers against each other showing simple pros and cons.same for software, storage and the rest."

Our response: Group tests are valuable but we want to have the detailed research to support research that provides novel insights rather than simply rehashes marketing specifications. As we finish up testing products of a certain category we will release group test results.

Reader comment: "Maybe organise (paginate) each review for consistent layout and points covered, for easier quick product to product comparisons of your reviewse.g. http://techreport.com/articles.x/17402 (see bottom for pagination, this layout is often used for IT product reviews)"

Our response: For cameras and VMS we do have standard organizations (though we lay out the complete review on a single page (12 page reviews often do this to maximize advertising revenue - a concern we do not have). For cameras, it's usually (1) overview, (2) pricing, (3) physical, (4) configuration/optimization, (5) image quality, (6) recommendations.

Reader comment: "1. Examining the company and its product within the targeted marketplace to understand overall positioning and chances for success.  I think you do this REALLY well, John, and your insights show your many years of experience.2. Real-world testing.  This is an area that's SO difficult to cover, especially in, for example, the area of analytics testing to determine reliable and replicable metrics around false positive mitigation AND positive detection in multiple scenarios.  For example, I believe perimeter detection/verification is a complete application in itself:  there are specific capabilities (and economics) required there that may not be applicable in a generic Public Safety application. Anyway,  I think it requires more resources (people, locations, time) than you may have--and am wondering if you should hook up with some independent company that may have more infrastructure."

Our response: We could certainly work with other companies to do test. Ultimately though something must cover the costs. If we partnered with another company, either we would have to generate more subscriptions, charge higher prices or have the other organization fund it (in exchange for?). We are a growing, profitable business and plan to continue to expand the depth and sophistication of our research and testing. We think this is the best way to remain independent and true to the needs of users and integrators of products.

Reader comment: "I value your research but wish the subscription model was more affordable.I feel like we end up being unpaid beta testers for every new product out there. Mfr says it is ready for deployment and we end up pointing out all the shortcomings during the install - on our time."

Our response: We agree that a lot of integrators wind up being unpaid beta testers. This happens to us frequently. Our results help integrators and users avoid bad decisions which can easily save far more than the cost of our subscription (about the price of a cube camera for a year). We'd be happy to charge whatever rates that provided sufficient revenue to cover costs. However, independent research that is dedicated to finding flaws need to be paid for by users and integrators. The service can be made cheaper by getting manufacturers to fund but I can assure you that the value of the service would collapse as manufacturers pushed for more favorable treatment. That being said, we are happy to entertain other ideas on pricing that does not resolve around manufacturer funding. Also, see our subscription preview to see the value the service provide.

Conclusion

Related Reports

Manufacturers Not Interested in October 2020 ISC West Show on Apr 13, 2020
Manufacturers overwhelmingly say they are not interested in going to the...
Fever Camera Sales From Integrators Surveyed on Jun 01, 2020
Fever cameras are the hottest trend in video surveillance currently but how...
Facial Recognition: Weak Sales, Anti Regulation, No Favorite, Says Security Integrators on Jul 07, 2020
While facial recognition has gained greater prominence, a new IPVM study of...
Integrated IR Camera Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Uniview, Vivotek on Jan 30, 2020
The best and worst cameras tested in this IPVM shootout showed major...
Faked Coronavirus Fever Detection, Athena Used Hikvision; Responds - Selling NDAA Compliant Cameras, Pledging 50% Of Profits to Victims on Mar 24, 2020
US company, Athena Security, faked its coronavirus fever detection marketing,...
Spectron IR Thermal Fever Screening System Examined on Apr 14, 2020
Most are quick to avoid "fever screening" and "medical" labels, but...
Masks Cause Major Facial Recognition Problems on Feb 24, 2020
Coronavirus is spurring an increase in the use of medical masks, which new...
The Problem With Fever Detecting Thermal Sunglasses on Apr 15, 2020
While the media has promoted using thermal sunglasses to detect fevers, this...
Manufacturers Doing Better Than Expected Against Coronavirus on May 05, 2020
Coronavirus impacts are not hitting manufacturers as badly as they feared,...
Dynamic vs Static IP Addresses Tutorial on Apr 16, 2020
While many cameras default to DHCP out of the box, that does not mean you...
FDA Gives Guidance on 'Coronavirus' Thermal Fever Detection Systems on Mar 30, 2020
The US FDA has given IPVM guidance on the use of thermal fever detection...
Integrators Avoiding Coronavirus Air Travel on May 29, 2020
IPVM asked integrators if air travel is part of their 2020 plans to see how...
Pivot3 Mass Layoffs on Mar 27, 2020
Pivot3 has conducted mass layoffs, the culmination of grand hopes, a quarter...
Vulnerability Directory For Access Credentials on Feb 20, 2020
Knowing which access credentials are insecure can be difficult to see,...
The US Fight Over Facial Recognition Explained on Jul 08, 2020
The controversy around facial recognition has grown significantly in 2020,...

Recent Reports

Dangerous Hikvision Fever Camera Showcased by Chilean City on Aug 07, 2020
Deploying a fever camera outdoors, in the rain, with no black body, is...
"Grand Slam" For Pelco's PE Firm, A Risk For Motorola on Aug 07, 2020
The word "Pelco" and "grand slam" have not been said together for many years....
FLIR Stock Falls, Admits 'Decelerating' Demand For Temperature Screening on Aug 07, 2020
Is the boom going to bust for temperature screening? FLIR disappointed...
VSaaS Will Hurt Integrators on Aug 06, 2020
VSaaS will hurt integrators, there is no question about that. How much...
Dogs For Coronavirus Screening Examined on Aug 06, 2020
While thermal temperature screening is the surveillance industry's most...
ADT Slides Back, Disappointing Results, Poor Commercial Performance on Aug 06, 2020
While ADT had an incredible start to the week, driven by the Google...
AHJ / Authority Having Jurisdiction Tutorial on Aug 06, 2020
One of the most powerful yet often underappreciated characters in all of the...
SIA Coaches Sellers on NDAA 889B Blacklist Workarounds on Aug 05, 2020
Last month SIA demanded that NDAA 899B "must be delayed". Now that they have...
ADI Returns To Growth, Back To 'Pre-COVID Levels' on Aug 05, 2020
While ADI was hit hard in April, with revenue declining 21%, the company's...
Exposing Fever Tablet Suppliers and 40+ Relabelers on Aug 05, 2020
IPVM has found 40+ USA and EU companies relabeling fever tablets designed,...
Indian Government Restricts PRC Manufacturers From Public Projects on Aug 04, 2020
In a move that mirrors the U.S. government’s ban on Dahua and Hikvision...
Directory of 201 "Fever" Camera Suppliers on Aug 04, 2020
This directory provides a list of "Fever" scanning thermal camera providers...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Dahua Loses Australian Medical Device Approval on Aug 04, 2020
Dahua has cancelled its medical device registration after "discussions" with...
Google Invests in ADT, ADT Stock Soars on Aug 03, 2020
Google has announced a $450 million investment in the Florida-based security...