Large End User DIY Surveillance - Recommendations?

Would appreciate recommendations or shared experiences on staffing levels for video surveillance installation. This would be for 500 cameras with SAN storage, and three camera vendors including Areconts. We are considering insourcing support due to poor VAR experience. Would involve the following duties:

  • Sys Admin of SAN, Servers, and VMS software
  • Perform and coordinate VMS software upgrades with Camera firmware upgrades.
  • Troubleshoot camera issues and possibly replace cameras.
  • Annual testing of redundancy.
  • Monitor resource usage such as disk and CPU.
  • Vendor Liason or possibly Liason with VAR who may stay involved as backup.

If anyone has gone through this process of insourcing, that would help as well because there are all sorts of other questions to consider such as licensing and support costs with the vendors.

Very interesting question, one notice to integrators: Do NOT criticize or disagree with this end user's decisions. If you want to debate the merits of this approach, feel free to open up a different discussion. This one will just be about helping this end user.

Some initial questions / recommendations:

  • Has the system already been deployed? In other words, does this include design and/or implementation or just maintainence?
  • Do your internal people have any training on the specific cameras or VMS software you have? If not, I think that would be useful, even if it's just a 2 day class to go over the specifics of each.
  • I would try to learn as much as possible about what monitoring / maintenance functionalities are built in to the VMS. This would be the easiest and most straightforward way. The potential downside is that functionalities might be missing (depending on the VMS).
  • Have you checked if all of your vendors allow buying direct and/or through the Internet? Most do, including Arecont but some VMS vendors may refuse to support or sell you licenses directly.

<snip>Do NOT criticize or disagree with this end user's decisions. If you want to debate the merits of this approach, feel free to open up a different discussion. This one will just be about helping this end user.<snip>

Thanks for that John. Almost every forum I've ever been on needs this sort of statement at least from time to time if not more often. That's just awesome.


Casinos are known for this (at least some of us...), though we do hire Integrators on occasion and will be hiring one for our complete VMS/NVR system replacement - we don't have the staff or sufficient knowledge/training to feel comfortable with getting the changeover accomplished expeditiously.

That said, we install every camera; pull and terminate every cable, perform system programming, upgrades and maintenance with a staff of four: myself (System Administrator, designer and troubleshooter), my Lead Technician (second Administrator, designer and Installer) and two line technicians.

In actuality, with appropriate training on the chosen new VMS, we could design, install, program and maintain the complete 1,200-channel system ourselves but probably not in the time frame we are contemplating a full service qualified Integrator could accomplish it in. In our case, downtime must be kept to an absolute minimum and that takes more qualified people plus a few competencies, like network programming, that we lack.

If I had a CCNE and a couple of extra trained techs (at least during the install phase), I would seriously consider tackling the entire job in-house.

Carl, thanks. How do you deal with integrators who want more or want to sell you products? Do you simply DQ them and show the door? Have you had any issues finding the right talent with this approach? I could imagine some integrators, especially large ones refusing. I am curious how you navigate around that.

All good points and there are several markets that a lot of in house moves,adds and changes are performed. Healthcare, higher education and government to name a few. I would add that you want to be up front with all you discuss with. They will undoubtedly want to gain your business at some point. If they perform well you just might want them too. I would be scolded for saying this but even quality VMS partners haven't encountered every possible scenario. It would behoove you to have competent in house people for any contingency. Manufacturers will work with you as well but it is case by case and you can't come across as the boy who cried wolf. Document your reasons for wanting to stray from a certified VAR and stand firm ground. As a large integrator I welcome your assistance in install and service. I can't be there 24/7 at low cost, but I can support reasonably and provide training and guidance to ensure the solution is a glowing example of partnership. That is a best case win/win.

A number of my clients use in-house technicians to service and support their security and surveillance systems. There is usually a much greater effort in installing a system than there is in maintaining it, and for this reason outsourced rather than in-house labor is usually used for the initial install. While some users may do some minor installation work (such as additions to existing systems), most hire integrators to do the installation and use their in-house techs to service and support the system afterwards.

It is essential that the your desire to service and maintain the system yourself be well-known by everyone upfront. Your bid specifications should state this explicitly and manufacturers should commit to providing direct technical support to your techs without requiring that they go through an integrator. Some manufacturers are good about this and may even have special support programs for end-users, others absolutely forbid it and require that everything go through an integrator. This is especially frustrating when your techs are more qualified than the integrator's techs. In these case, the integrator can actually serve as an obstacle and source of misinformation rather than as a source of help. (in-house techs who work on the same system day after day can get REALLY good..)

I have often seen end-users hire their security/surveillance techs away from integrators, perhaps even from the one that did the original installation. I won't comment on the ethics of this, but it is one way of recruiting a qualified technician. Many techs prefer the stability and benefits that can be provided under such an arrangement and jump at the opportunity.

John, Never really thought of it. We did have a few who approached us in that vein - they were shown the door. Some Integrators specialized in systems we rejected in our Phase 1 tests. Those were also eliminated. We had three systems proceed to Phase 2 testing, along with three Integrators. One remaining Integrator specializes in only one of the three remaining systems, although the other two Integrators do integrate that same system. One system only has one Integrator for our region. The remaining system can be integrated by the other two Intergators.

I'm confident that any of the three remaining Integrators would perform the tasks more than adequately. They all have a large presence in the casino industry and many recommendations.

With the customers we have that for the most part are self sufficient for maintenance, a rough staff count of 1 full time experienced tech per 250-300 cameras seems manageable. We will still typically augment them for MAC work, server upgrades and system growth architecture. It's not rare in the K12 and higher ed markets from what I've seen.

Thanks Scott, and thanks everyone else who weighed in on this! Let me add a few more assumptions:

We would continue to outsource new installs for now. We would also outsource major upgrades with the exception of incremental software upgrades. Another assumption is that we would plan on sending the tech to any required vendor training. We already have very experienced network engineers on site who help with that when necessary. We also have others who actually do cabling and networking installs. We also have people who know SAN and server technologies, but we would expect the new hire to become the expert in the flavor of SAN that typically is used for CCTV.

Let me also clarify that this is an existing installation that is currently being supported by a VAR, but has been a bit of a disaster.It is unclear at this point how much of the problem is attributable to the VMS vendor. (NICE). If we take it over, we can at least take the VAR out of the equation and make that determination.

We may also continue to use a VAR to cover vacations etc. if that is possible.

My hope is actually that we could get away with closer to .5 of an actual FTE for day to day maintenance. Once the dust settled, say after the first year, we would consider looking at the econmics of doing our own installs as well.

Thanks again.


That explains it. I can't put this any more succinctly: NICE sucks!

NICE bought the manufacturer of our current system, Fast Video Security, AG, in 2005. After the purchase, system development petered out as programmers quit or were laid off until by 2011 there was no further development at all. If you go to the ExtraNice website now, you will find that nearly all files have been removed. That includes all but the final software version (6.00.18) and one version of their encoder firmware. Even their Technical Notes are gone. It's lucky I saved many of them on my PC.

The last software release was in April of 2011. Without new releases, no new IP devices are supported.

Lack of development and support is the primary reason we are replacing the entire system instead of just updating the hardware. And their North American vendor is no better. Honeywell still sells the product (known as Honeywell Enterprise) despite the lack of developers.

Tell us how you really feel. :) I do appreciate the candor, and would love to hear more opinions about VMS vendors as we have seriously considered switching. Easier said than done though, partly due to lack of standards for cameras.

I should clarify that we did upgrade to NiceVision 2.5 something. It did apparently make some admin tasks, but we still have lots of issues with cameras going "dead" and other various problems. One of my thoughts actually is that if we did convert to another VMS vendor, we would the tech's job would be easier once over the hump.

Maybe this is another whole thread, but I'm wondering what is the amount of cameras per 500 that should just go dead say per month? This assumes that the camera can be revived through reset or some other process. More than a dozen?

Hell hath no fury like a Casino Security Director scorned...

Surveillance Technician Manager, but I won't dispute the fury part...

Seriously, we were approached by the NICE rep about converting to NiceVision. I told her there was no way in hell. I had approached NICE many times over the years due to what I felt was a lack of support by Honeywell. I was rebuffed each time with the statement that Honeywell is the sole U.S. representative for casinos. Honeywell is another company on my sh@# list. I wouldn't buy their products if they were the last company on Earth. They don't have a clue about what they sell.

And the family relationship between Honeywell and NAV assures NAV they will be the sole supplier there. Andrew Jackson, the head of Honeywell Video sales, is the father of Daniel Jackson, head of NAV sales. Try to get pricing on an encoder or other part from any other source. You'll pay at least 20-25% more because of that sweetheart deal.

And even though they switched encoders from the Fast ENC8 to the NICE-manufactured NVE1008 when the parts became obsolete, you can't buy the NICE encoders directly from a NICE vendor if you own a NICE/Fast system. The firmware is different and they won't sell the Fast firmware for the encoders. We had the opportunity to purchase many NICE NVE1008 encoders for <$1,000 each that were "lightly used" when Alabama shut down casino gaming. Honeywell gave us a song and dance saying NICE wanted as much for the firmware as the cost of a new Honeywell/Fast encoder (~$3,200 for 8 channels).

Even though NICE stopped development on the system in 2010/2011 (the last release was v6.00.18 in April of 2011), the last four version releases had severe bugs that were never corrected. After the last "stable" version, v5.75.19, every other release experienced compatibility issues with a PCI card they use to provide an analog output from the client (a so-called VO1 card). Our clients that have that card often require a full reboot before a new user can log on and often crash the program.

Our servers must be rebooted approximately every six months to avoid locking up. They are running Windows Server 2003, which was very stable on v5.75.19.

Analog or IP? From my experience in a ~1,000-camera environment, we have maybe one or two go bad per month, but they are typically around 10 years old. Newer cameras rarely fail. For instance, in the last 1-1/2 years, we have installed well over 200 inMotion in11S3N2d cameras and over the past 4-5 years, another 200+ Vitek VTD-A2812. Of the 400+ cameras, we've experienced "0" inMotion failures and only 2-3 Vitek failures. In fact, we are still tunning some Pelco CCC1370-h2 cameras installed in 2002-2003 and some Ganz and Ikegami cameras that are in the same age range.

Obviously, all of the above are analog. With IP cameras, YMMV (heat and all).

If you're using analog cameras with NVE1008's, all I can say is that those encoders also suck - at least when coupled with the Fast VMS. Losses of power cause the encoder to change bitrates, GOP type and/or length and even enable secondary channels that were disabled. If we have to reboot an NVE1008, we always have to go through all 8 channels to make certain any settings didn't change.

If it's a dozen consistenly a month, that's 144 per year or 28.8% of all cameras. That sounds like a hell of a lot, unless the cameras are really old (7+ years) and/or in very adverse environments or you're using Arecont cameras (kidding somewhat).

I'm talking IP cameras.

Do they start working after a reset?

Sometimes. Other times the problem is much more complex to fix. BTW, this does not include other intermittent problems, image view problems etc. which would add to the number.

Do you see any patterns? Is it disproportionately in different areas of your facility (indoors vs out)? Is it skewed for one model vs another?

Spread across Arecont, Honewell and Sony.

temperature? power? connectivity issues?

We spent many, long hours proving that there were no power or networking issues, only to find that in the end there were firmware incompatibilities causing many of the problems. It is clear that some CCTV Vars don't know what they don't know about things like networking and PoE.

I would seriously consider replacing NiceVision. Our NICE/Fast/Honeywell system shares many features with NiceVision and it also doesn't play well with IP cameras. As far as I can tell, NICE shared at least some of the code between the two systems after they purchased Fast. In fact, that appears to have been what their intention was when they purchased the company - strip it of everything of value and hang it out to dry.

In my experience, about 1 in 5 larger school districts and other educational customers seem to prefer taking as much physical security work as they can in-house. The best example i know is the 3,000-camera 'take over' for the Austin ISD (Texas) where they converted DVTel to Video Insight VMS. The customer knew, as many others now do, that they could better manage their budget and utilize their staff by doing all they could in-house, and contract the special bits (camera locations for new buildings, Q&A for camera recommendation, etc) to third-parties while performing the IT and Facilities parts themselves.

Seems to me that the DIY trend is growing as IP systems become prevalent. What is the group seeing?

We're an integrator who supports a school with around 500 cameras. Most of what we do is act as the re-seller since the VMS and camera vendors do not sell direct to them as an end user. The school obviosluy has their own IT staff from bench techs to sys admins and they did the installation and initial setup on their own. They had a couple of their techs go to training classes for the VMS and the camera manufacturer they used.

We spent a couple days assisting them with the initial configuration since it was a new system to them. And it was good because they missed some obvious errors in their VMS settings for recording and viewing. Then they had a problem with their iSCSI storage units which turned out they were trying to push connections between the VMS and SAN's with a 100mb switch. (I kid you not. They had been mis-informed by their network guys what kind of switch it was.)

With their own staff it made sense for them to do most the work themselves. But were smart enough to use us for assisting in the initial configuration and setup.

My adivce would be to have at least one person on your staff whose primary duty is the video system. They can have other duties, but his/her first responsibility is the video system, they get to know it and learn everyhting about it by practicing on it regulalry. And as someone mentioned on here earlier, they more they use it the easier it will become. It's like anything else. If you have a major system running on an in-house SQL server, you have someone who knows at least something about maintaining an SQL server. They don't have to be a database architect; they just have to at least know how to backup, run maintenace, and troubleshoot client connections to tthe SQL server. Same for the video system.

And whoever the integrator is you use for fall-back support, make sure they have someone who has experiance not only in security but also some experiance in the IT world, 5 years or more preferably. That way they better understand your concerns and challenages and can speak your lingo. And trust what they say, especialy when they tell you those name brands in the IT world don't mean they'll work that well in the security world. It's not quite like any other IT system, but it's simlar enough you should be able to do most of the work yourselves if you properly allow your staff the time and resources to get to know it.

Undisclosed integrator, great feedback, thanks!