Subscriber Discussion

Setting Up A Camera Shootout? Good, Bad, What Issues?

U
Undisclosed #1
Jul 13, 2017

Have you ever participated in a bid where competing camera manufacturers were scored by the client on the best picture in a side by side comparison? As John has noted/inferred  in previous articles, quality is a hard thing to specify. Lets say its a blind comparison. Monitors are situated in front of the customer (which by the way includes his IT guy who is interested in things like storage, bit rates, etc). Monitors and workstations are provided by the IT department, all of which are the same. The customer evaluation team consists of the aforementioned IT guy, police or security personnel, physical plant and the Project Manager.  The team is given a score sheet upon which they are to rate on a scale of 1-10 the best (in their opinions) picture in several controlled lighting situations and all cameras are viewing the same scene.    

Note that this is only one of the many factors in evaluation of the system overall.  

The object of the game is to choose the camera only from among those proposed by the integrator. Or, in another situation, the highest scoring manufacturer would be specified for the project. 

I have camera manufacturers tell me all the time they would be happy to participate in such a gig. 

What do you think? What changes would you suggest? Have you ever gone through this exercise?

 

 

JH
John Honovich
Jul 13, 2017
IPVM

Good question. A few recommendations:

  • If you let manufacturers change settings, track exactly which ones they are changing / setting. If you let them change things however they want and not let you know, you will have no idea whether it was fair or rigged, etc.
  • Make sure to continuously check bandwidth throughout the test. For example, if the manufacturer knows you care most about quality, savvy ones will lower compression to give an edge hoping that you do not realize the bandwidth impact.
  • Make sure to do tests in hard conditions / times (e.g., no moon or setting sun or fog or whatever environmental issues are material to one's facilities). The easier the conditions, the easier it is for all cameras to look the same. Harder conditions will better show differentiations.
  • Be careful about letting manufacturers change settings during the test. In real operation, you will not 'tweak' the camera multiple times per day. However, an experienced manufacturer can give their cameras an edge by knowing what settings to change. However, this is not realistic as in production you won't have a person manually changing settings like that.
  • Lastly, and again related to settings, compare what settings the manufacturers are using. Differences in performance could again come down to just differences in configuration vs differences in actual product quality. For example, if you don't care about motion blur but want real bright images, than all cameras should be set to a slower shutter. If just one manufacturer sets the shutter slow, that is not a benefit of their product, it is a temporary benefit of that one manufacturer's setting change.

I think on-site customer shootouts can be useful but they should be managed / controlled properly to ensure that you are picking the best product, not the best salesmanship / gamesmanship.

(3)
UM
Undisclosed Manufacturer #2
Jul 13, 2017

Make sure ir is turned off in all camera, since you could have ir from camera 1 interacting with camera 2, etc. Try to record for 24 hours to get a range of lighting conditions if it is outdoors or outdoor facing. 

Make sure osd is turned off in the camera as to not give away the manufacturer. 

Make sure to take image snapshots and zoom in to details.  

You may want a resolution and color test chart to provide quantifiable details to analyze. 

(1)
(1)
JH
John Honovich
Jul 13, 2017
IPVM

Make sure ir is turned off in all camera, since you could have ir from camera 1 interacting with camera 2, etc

To #2's point, shootouts are now a lot more complex than they used to be with most cameras shipping with integrated IR. So if you are shooting out 4 IR cameras simultaneously, you will have effectively 4 IR 'lamps' shining at the same spot, which will screw up the results, making bad low light cameras look good or good ones look washed up because of too much 'light', etc.

Logistically, this is problematic. To be truly fair, you need to make sure only one IR source is on, which means as #2 says, either turning off IR in all the others or turning off all other cameras entirely so they are not in any way judged off the IR from another camera.

(1)
UI
Undisclosed Integrator #3
Jul 13, 2017

Many times as a manufacturer to support dealers.  

Allow time for each manufacturer to configure their own cameras, let them succeed or fail.  Although IPVM tests "out of the box default", manufacturers will scream "unfair".

One challenge is night viewing with IR.  You can't leave the IR on for multiple cameras at the same scene and cameras without IR will benefit.  The results will be unrealistic. 

The test should run for several days. 

Historic flops:

Brand A failed miserably in a night time shoot-out because the sales person wasn't technical who attended and next to them was a camera with strong IR reflecting into the dome.  I looked at the image and it really was bad and did not represent Their actual imagery.  I sat quietly in the corner.

Brand X lost because the dome camera was mounted on a board sideways and there was reflected light from a nearby fixture.  All of the actual cameras would have been horizontally mounted.

Brand Y lost in a shoot-out because the sales guy didn't understand the recorder he plugged his multi-megapixel camera into was using VGA to the monitor and that his chosen recorder, for simplicity, was limited to 800x600 on the output  

 

(1)
JH
John Honovich
Jul 13, 2017
IPVM

Allow time for each manufacturer to configure their own cameras, let them succeed or fail. Although IPVM tests "out of the box default", manufacturers will scream "unfair".

If you let manufacturers configure their own cameras, you are likely to buy from the savviest salesman, not the best product. Oh... now I understand what you are saying ;)

In all seriousness, I cannot strongly enough urge end users and integrators who run shootouts to carefully track and consider every setting change the manufacturer makes, as they often create unforeseen side effects that the manufacturer will be happy to hide until after the deal is closed.

(1)
(1)
UI
Undisclosed Integrator #3
Jul 13, 2017

John,

For the most part I agree with you.  I certainly understand the points made.  I expect IPVM to run generic default tests and too bad for the manufacturers that haven't spent the time to modify their defaults to best position the product.

Years ago I fought and lost a battle with product management as we released and continued to sell a camera that cost more than our competition because of a specific feature we DISABLED in the default settings when shipped!  I had to go to several dissatisfied customers and enable it to make them happy.  

The reason cameras have options is to suit the possible installation variables.  It's unfair not to take advantage of them and only fair if you allow the manufacturer to adjust them. 

Documenting test settings is a great idea.  If I were configuring for a quick shootout I could certainly tune my product to just that test and it would be misleading.  

Every time I see a manufacturer use a still image to compare low light capability it actually angers me. 

If the test runs for a few days and they use recorded video for the final evaluation then adjusting "for the moment" will be detrimental to that manufacturer.

Suggestions of turning off all text and such is valid.  IR conflicts are surprisingly missed.  Data rates will be captured by the recording as well as how those changes impacted the final product.....a recorded image.  

My suggestion is "Judge the final product" based on moving video clips across several time periods and the storage required to obtain them".

U
Undisclosed #1
Jul 13, 2017

I would want manufacturers to be allowed to tweak the performance of their camera and I can see why manufacturers don't like out of box performance tests. IPVM is the exception however, because you have the folks with the skill sets to qualify a review. 

Also, many of these jobs are public, and I take great pains to make sure everyone is satisfied. In fact, i would ask those supplying the cameras to voice their opinions before the shoot out was set up. 

U
Undisclosed #1
Jul 13, 2017

John. Can you tell me how you do that left vertical line thing for quoting responses?

UI
Undisclosed Integrator #3
Jul 13, 2017

That's like unmasking Batman ;)

JH
John Honovich
Jul 13, 2017
IPVM

John. Can you tell me how you do that left vertical line thing for quoting responses?

Copy and paste the text. Then select the blockquote icon, as shown below:

U
Undisclosed #1
Jul 14, 2017

I tried this many times before but did not realize that it only magically appears when you post it... Now your secrete is revealed!

(1)
UI
Undisclosed Integrator #3
Jul 14, 2017

AKA Bruce Wayne unmasked. 

U
Undisclosed #1
Jul 13, 2017

Other than band width and storage, what are these side effects ?

UI
Undisclosed Integrator #3
Jul 13, 2017

Recordings will tell you if they increased the bitrate to the maximum to improve recorded video, but didn't disclose how much additional storage would be required. 

As for the actual image, every action causes a reaction.   IPVM tests highlight when a feature isn't a feature if improperly applied. 

The most basic is making your night images look great by increasing frame integration and slow shutter to such a point that anything moving slowly is blurred, resulting in a pretty still and useless security video.

In the old "box camera" days, you could put an $800.00 lens on a camera to improve the results in low light while specifying a $80.00 lens in the project.

I like it when car companies have to tell you "with additional options installed" as they show the most tricked out model while stating "economically priced"

JH
John Honovich
Jul 13, 2017
IPVM

what are these side effects ?

A big side effect is hidden bad performance. A combination of settings can look great for a specific scene/time but it may be terrible for later in the day or night or when the weather or lighting changes. So if you let a manufacturer choose their own settings you need to determine the image quality of those settings over the course of a longer time frame.

U
Undisclosed #1
Jul 14, 2017

A big side effect is hidden bad performance.

Yes, but the test would be the same for everyone. Same lighting, same target, same movement in the scene.  Point is that we are comparing one camera against another. I would also listen to the manufacturers to make sure they feel the test is fair.  What do you think of having manufacturers view their competitors?

 

(1)
U
Undisclosed #1
Jul 13, 2017

Good points all. There is a balance between bandwidth and image quality. The test would include allowing the manufacturer to tweak the camera for best picture and least bandwidth with each representing half the score. Frame rates would be fixed at 24 (or any number for that matter as long as all the manufacturers can go that high). Or maybe images could be taken at a range of settings. Its only important that the variables be the same for all the cameras involved.

Regarding the non technical sales guy doing the shootout. If i am doing a 500 camera system I would think a manufacturer would send a person with a higher level of skill. Perhaps the RFP could require it.

UI
Undisclosed Integrator #3
Jul 13, 2017

Where do you draw the line?  

I saw a small test represent a very large sale, unknown at the time by everyone.

Offer any manufacturer that would be considered a reasonable amount of time to participate, or not.

First there needs to be an evaluation of manufacturers to see if they, as a whole, have the necessary solutions and meet the end requirements if the test is successful.

That usually runs it down to about 5 and they aren't always the same.

U
Undisclosed #1
Jul 13, 2017

I agree about where do you draw the line. As JH has pointed out many times, image quality is in the mind of the viewer. But the needs vary from one project to the next. Had one job where there were 10 cameras on the exterior and over 200 on the interior. In this project, we chose different manufacturers for each situation. 

Regarding the evaluation of manufacturers, I have 10 on my list that "could" meet my loose set of criteria  (years in business, support line, documentation, etc) for government projects. For non state projects I would probably limit this number to 5 just  to save time. 

(1)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions