What To Do About Mission Critical Failures And Utter Disarray In Security Projects?

A manufacturer offered an insightful comment about serious problems in major security projects:

"Having been intimately exposed to the shortcomings, unkept promises, mission critical failures, and utter disarray that has all too often been described by end users with respect to CRITICAL INFRASTRUCTURE (and I don't use that term loosely), or even just a 20 camera system at a middle school, frankly, it is sometimes startling."

What do you think? Is this a real issue? If so, what can be done to deal with it?


A two-pronged approach is required:

  1. Learn, learn, learn! As an end user, I've found you can deal with vendors from a position of strength if you have the knowledge to talk with them at their level. It also helps avoid BS.
  2. Don't be afraid to use that strength of knowledge to your advantage and to be a "squeaky whell" when necessary.

In terms of learning and knowledge, you are the exception rather than the rule, both in depth and length of experience.

Who's the prototypical security buyer? In my interactions, whether they come from security, facility or IT, they usually only have limited skills / experience dealing, testing, working with surveillance products. I am not criticizing them for it. They typically are managing a broad array of programs, technologies and issues so it's unrealistic for them to be experts in every specific area.

Carl, throwing the question back to you. Do you think this is a fair description of many security buyers? And for those who fall into this situation, what can they do?

Others' laziness or lack of knowledge in not my concern. It goes back to my point about excellence in another thread. If security buyers don't make the effort to at least learn enough about what they are buying and using to be able to ask the right questions, provide the right specifications and/or recognize BS, they have no one to blame but themselves when things don't live up to their expectations.

Whenever I encounter a new situation or confront a new technology, I take the time to familiarize myself with it; at least enough to make intelligent decisions regarding its deployment. At the very least, I make certain someone else in my sphere or who I know and trust has that knowledge.

Carl, it may not be your concern, but it is the concern of this discussion. It it's not important to you, simply refrain from commenting on this.

It's hard for people to learn all the nuances of a domain. Think about all the schools and cities that have to buy surveillance. It's genuinely hard, even if they are responsible and determined to make an informed decision. And even for people that are lazy, it is still an industry wide issue of how do we help prevent / stop such nightmare projects?

This is a real issue, and the one that made me leave the systems integration business and start my consulting practice in the first place. After more than 25 years as a consultant, I still don't have all the answers, but some of the major contributing factors to troubled projects I have seen include:

  • Failure to clearly identify goals and expectations at the start of the project.
  • Failure to clearly communicate these goals and expectations to all involved parties at every stage during the project. Biggest failures occur at points where project is handed off between parties (customer-to-salesperson, salesperson-to-engineer, engineer-to-project manager, project manager-to-technician, etc.)
  • Unrealistic deadlines/completion schedules.
  • Lack of independent project oversight by a technically capable party.
  • Lack of staffing/human resources to properly manage the system once it has been installed.
  • Person who must ultimately use the system is not adequately involved indesign, planning, and installation of system.
  • Specifying features and capabilities that don't exist in the marketplace, assuming that they will be created specifically for your project, and then expecting that they will work flawlessly when first implemented.
  • Expecting manufacturers to eagerly embrace system integrations when they have little or nothing to gain from working with competitors.
  • Paying contractor based on materials and labor supplied rather than on demonstrated completion of systems.
  • Failure to use professional trainer to provide formal training sessions to customer employees.

With more systems being IP expect it to get worse. We import products from China and test and inspect everything before buying. Our reject rate, after filtering the realy bad at sample product level is around 80%.

Problems like Sony being in fact Sharp, and have even seen chips skimed and reprinted Effio-P, Lens full of dust, Its a high cost having an office there but we check everthing before and during buying. We save a lot of problems later.

Hire Experienced PMs, Professionals with Talent, Project Experienced Individuals, More Accountability, Pressure to perform and Rewards for performance, Bonus's for success and milestone achievements, Progress Bonus's for ontime, ontarget, successfull milestones

Improve Contracting Standards & Hire Consulting Teams for better oversight and enforcement of Codes, Standards, Practices .

Better Planning Teams with Experience ( not just degree's)

This is a serious problem in our industry mainly in the gov. sector as they give contracts out on concept and not alway s proven workmanship.

A lot of projects are managed by Civil, Mechanical,Geo engineers which have no background in security or survaillance .

I have worked on many projects which were not set in stone only concept.

Engineered, Designed, Conceptualize the details out as the process evolved.

If the project went bad you just redirect, rework the concepts untill you succeed.

Common in Large Projects which do not have set standards. Congressional Authorized .

Design Build is a good word for this . Not What s on the table , but where you can take the project and prove you can get it done with in listed parameters .

Impress the management team

Have a Budget and start building the concept.

Usually these are very costly and overruns are common.

usually 30 to 80% overrun in cost. 1.5 mil start , when finished around 5 mil.

In the last 10 years with the updates and upgrades of software packages, you dont see this so much. Easier to Build Models , 3-D Design Modules , and Micro model the projects.

Imputting Data just got easier and more user friendly

Great Cost Analysis software ( Time,Install,Materials,Quanity, Margins, Pricing, O/H,Profit )

Thank You MS, Autocadd and others

But when you work for some companies, they understand the whole picture and know you never have all the variables upfront, and you always open up a huge array of challanges in the process.

I now work for companies which have Zero Tolerance for Failure and Zero Downtime options with 100% reliability and Greater Expectation of Performance.

As a contractor , Not an employee this provides greater flexibility to get things done.

and also more pressure to perform

Failure is not an option.

This weeds out the rift raft. Limits the playing field

Good recommendations from Michael and Christopher. To these I would add:

  • Get good inspection and testing milestones into the project plan, so that problems are caught early when the cost to correct is smallest.
  • Perform proof of concept testing, prior to purchase and deployment, for features that are critical. Most "$1M-plus" projects that fail completely did not do this. For example an over-water video analytics at a seaport ($1.5M), a fence perimeter detection system at an airport ($90M). These could have easily been stopped or been changed to be successful following sound proof of concept test done before the project award or as the first step in the project.
  • Inspect progressively, as the work is being done, not all at once at the end. Many big project schedule-killing final-acceptance punch lists could have been eliminated by progressive inspection.
  • Do the user training as early as possible. "Project completion" by integrators means "acceptance of operational responsibility" by the end users, and "well-readied end users" should be an important project objective. Few things kill the excitement of project completion for the customer, than to have security personnel who are struggling to use the new technology, especially if it is a replacement or major upgrade project, and operator performance suffers due to unfamiliarity and "surprises" that features don't work in exactly the way they were expected to.
  • Define security operations scenarios (normal operations and incident response) to establish requirements for how the technology should perform. Base them on things that are easiest to do with the existing technology (don't lose current capabilities), as well as hard or impossible to do with the existing technology (maybe why they bought the new solution). Scenario-based requirements (concept of operations) are much better understood and much more useful than "the system shall" type of requirements.
  • Always be accurate and honest with the customer about project status. I know this is a hard one, as you can't be in a technology deployment business without being an optimist, and optimism seems to go into overdrive when things start to slip. I have seen projects big and small go sideways because an over-optimistic project manager reported "almost dones" as "dones" to the customer, and then couldn't catch up as more of these accumulated. Projects can hit "80% complete" and then stay there while lots of activity takes place but little progress that's visible to the customer, because the list of issues has been kept hidden. In ALL project situations I have seen, the customers would not have been anywhere near as unhappy if the small slippages had been accurately reported and even if the integrator had asked for more time. When the trust factor is strong (by accurate status reporting with periodic customer walk-throughs or inspections), customers can be very forgiving. When the trust factor goes away, projects can become living hell for all parties, and the integrator is then cut no slack at all. Plus referals are lost and customer bad-mouthing to vendors and project stakeholders can expected.