Major US City Satisfied With 20% Continuously Broken Cameras

Author: IPVM Team, Published on Jul 20, 2016

Is 20% continuously broken cameras reasonable?

Yes, it is, according to one major US city.

In this note, we examine the city, the challenges involved, and what is reasonable for broken cameras.

City Overview

A Dallas, TX city wide video surveillance system has received scrutiny following the brutal murder of 5 police officers. There have been problems getting video from the city's system (both in terms of quality images and sheer availability of any recordings).

80% Working

Dallas was straightforward about what was working and what was not:

80 percent of the 400 or so cameras scattered throughout the city -- cameras on which the city has spent millions in recent years -- are functioning properly.

With the chairman of Downtown Dallas, a group who contributed to paying for the system declaring:

Because of technology and manpower, you're never going to have 100 percent up at all times. Eighty percent is reasonable.

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

Old Milestone Mixed With OnSSI

An image from the control center shows that they use a mix of many years old Milestone and OnSSI:

For a major city to use such an old version, it is a signal that they either lack the money or organization to keep things up-to-date.

[Update: as a member mentions, this may be old OnSSI OEMed from Milestone, next to newer OnSSI, equally if not more dysfunctional.]

Firetide Failing

One other factor that likely contributes is Firetide mesh wireless. Back in Firetide's / mesh wireless 'glory days', this was considered innovation. But various reports of Firetide reliability problems in Dallas and Firetide's future slide into irrelevance changed that.

20% Not Working - Not Surprising But Not Acceptable

While we do not know all the specifics to comment on Dallas particularly, there are some factors that make this situation not surprising:

  • Overly ambitious system - especially using mesh wireless, which has caused reliability issues in many cities
  • Dated system - while Dallas was cutting edge 5 - 10 years ago when deployed, it is far from what systems can do today
  • Lack of upgrades - especially with city systems, money needs to be spent to keep the system upgraded and maintained or high failure rates are inevitable

One key underlying theme is that many cities get money to build a system (and in the past decade often from government grants) but then struggle to find the money to maintain, upgrade and refresh to keep it functional.

1 report cite this report:

12 Video Surveillance Poll Results 2016 on Dec 28, 2016
IPVM has conducted more than 100 polls this year. Here are a selection of 12 of the most interesting results, receiving 4,500+ votes: Hikvision vs...

Comments (24)

Only IPVM PRO Members may comment. Login or Join.

Couldn't that "old Milestone" actually be OnSSI NetDVMS running?

On another note, what is reasonable for a percentage of cameras not functioning in that environment?--mainly outdoor.

3-5%??

I don't mean this flippantly in any way, but 0% is reasonable.

Down for maintenance or brief issue? Sure. Down for extended periods?

5% of 400 cameras is still 20 cameras down at any given time, and that's horrible.

I don't mean this flippantly in any way, but 0% is reasonable...

Ethan's powerful "No cameras left offline" post has no doubt flooded his Inbox with numerous job offers from beleaguered CSO's around the world. Kudos.

I don't mean this flippantly in any way...

Day to Night, Top to Bottom, Arecont is the king of flippant.

Good question, I added to poll to see what people think.

Couldn't that "old Milestone" actually be OnSSI NetDVMS running?

Good eye!

My neighbor happens to be a Professor of Ancient American Recorder History at UCLA; he places it squarely in the Pre-Conversion Gigazoic period or older, and thinks it may even be a rarer SUP-less NetDVR 4.x specimen in the wild.

Couldn't that "old Milestone" actually be OnSSI NetDVMS running?

It could be. I don't know for sure. I do believe that the colors used to be the same years ago. For example, from an old IPVM test:

That said, to me, it would make less to see 2 different versions of any VMS running than 2 different VMSes simply because if you have a later version of a VMS, I would presume all cameras would run on that. Having 2 different VMSes, by contrast, is fairly common, if different times or organizations bought them.

...simply because if you have a later version of a VMS, I would presume all cameras would run on that.

Not recalling all the particulars, I remember being in a similar bind because I was SUP-less. And the licenses for the old NetDVR version were paid for, and as far as I remember you had to pay one way or another, e.g. a SUP on the old system or new seats on the new to get them all together.

Btw, this Firetide 2010 case study indicates I am wrong and they had OnSSI way back in 20120.

I still find it weird if they have 2 different versions running of OnSSI running side by side. I assume that's not OnSSI's fault but just weird and somewhat dysfunctional for Dallas.

I said up to 10%, but only because I think 1% is a really high bar. To me, I think it should be somewhere in between 1% and 5%, which leads me to voting up to 10%.

Maximum 5% down at any one time I think would be a more easier value to swallow assuming you put at least 95% effort into the maintenance and upkeep into the day-to-day care and feeding. If you are prepared to invest that amount of capital you need to UP FRONT also explain and get agreement IN WRITING from both management and unions to keep it running at those levels.

What is the annual maintenance cost of a city network with 400 cameras?

What is the annual maintenance cost of a city network with 400 cameras?

3, great question. The rough rule of thumb is 10-20% annual of install cost and that would put it at $1 to $2 million but I am not sure if that applies for this type of system (with heavy heavy infrastructure costs).

I suspect a key factor is where the outages are coming from - is the wireless system going down regularly? That could be very expensive? How many cameras are dying each year? If they are 5 - 10 years old, outdoors, it could start getting fairly high.

Yes, I estimated 15% based on the report you did on Baltimores system.

I couldn't imagine what NY or Chicago are paying per year.

Grants pay for everything up front, but the cities seem to forget the ongoing maintenance costs.

Grants pay for everything up front

Or did... There are certainly still grants but nowhere as much as the post-9/11 era where random cities could qualify for emergency anti-terrorism video surveillance system.

The goal should be 0% down or you shouldn't have put the cameras up in the first place. Things happen but you can get to 99%+ if you plan and resource monitoring, problem resolution and replacement costs.

If a camera is offline for a long period, it is because there aren't any perceived consequences for it being down. Either its because it wasn't needed or you've just been lucky up till now. It comes down to stakeholders and leadership.

Undisciplined operations don't suddenly turnaround at 21% failure. 20% failure begets 30% failure.

Boy, you sure are a hardliner! 0% is kind of hard to achieve in the real world when you get to this number of cameras. The Internet backbone is infinitely more reliable than any camera system buildout and even that is most likely five 9s at best.

If failure isn't an option, then why have plans or resource monitoring? It seems if you have a 100% system, end to end, you wouldn't need problem resolution or replacement parts.

While I'm not saying having 5% of your cameras offline all the time, I think having up to 5% down at any given time is a good worst case scenario. I can't imagine never having anything fail ever.

That's why I said that 0% should be a goal.

To operationalize it, you need to baseline your current performance and use data driven methods to improve operations. The fact that some of our customers achieve very high uptime (99%+) despite having 100's of video streams implies that they are motivated and resourced to achieve it.

With our system, they are alerted to every outage within minutes and respond. We have first failure diagnostics and analytics that make problem resolution efficient. A response can still be replacing equipment within weeks. These organizations that achieve high uptime at least initiate action on failure and resource resolution at a pace that makes sense but at least it is a process that completes.

At the same time, we have other customers who tolerate very poor uptime. Our monitoring solution can tell them when something is wrong and even point to root cause and suggested fix. However, they aren't resourcing the people to respond to address the issues or they aren't replacing equipment and systems as they run into trouble.

Budgets aren't there. People are overworked. Skills are lacking. As a result, people fight the fires they think are the most important and let the less important just keep burning.

Failures happen for two main reasons: configuration mistakes and/or deteriorating equipment. If you aren't aware of the failures or don't have a response discipline, things don't just magically improve when you hit 20% of your cameras offline. You need to have an operational plan and resources to respond. A shop that is accepting 20% failure is a shop that has 25% failure rate in its future.

Hi Jon,

To further clarify, if something like a server crash hits your infrastructure and you don't have a failover configuration, yes that can take out a good chunk of your camera streams all at once. At that moment, your uptime measure looks pretty bad.

However, to be effective, you need to trend uptime measures over some window of time, usually measured over the course of a month or quarter. If only 95% of your video streams are recording as they should over the course of a month, there are things that could be improved in your operation. You just need motivation.

Dave

Again, I never meant to say that 5% would always be down. My 5% cap was to mean at any given point in time, you should have a minimum of 95% of your system functional. More than 5% would be less than desireable to me. If you are talking about an average uptime over great lengths of time, then yes, 5% may be a bit high. Probably closer to 1%.

Let's define "Large system". Let's say it's >>250 cameras.

Let's define "non-functional". I think it means "no video and no explanation". I think if you had a huge system with complex challenges you could have up to 10% "non-functional" but I would like to think most of those have trouble tickets filed. I think you should have 5% or less "non-functional" where that's news to the system operator.

GE ran MobileView in CTA in Chicago will less than 1% failure at any given time I believe. We run a large school systems bus video and have less than 1% after they let us spend more on the install do to it right. The school stated that every incident is worth 1 Million so every camera in every bus had better be working. The entire system serves no purpose if you constantly have product failing. At the same time, the client has to pay to have it done right and then have a reasonable maintenance budget.

If you are doing remote guarding Misses of 20% due to just camera failures is unacceptable. One should also check (at night) scene illumination, camera view, analytics ROI and ROD on a periodic basis (nightly and weekly) depending on site type.

Related Reports on Wireless

Access Control Cabling Tutorial on Jan 15, 2019
Access Control is only as reliable as its cables. While this aspect lacks the sexiness of other components, it remains a vital part of every...
Winter 2019 IP Networking Course on Jan 10, 2019
Today is the last day to register for the Winter 2019 IP Networking course. This is the only networking course designed specifically for video...
European Startup Ajax Profile - They "Stand Against Evil" on Jan 03, 2019
European intrusion detection startup Ajax Systems proclaims: How are they standing against evil? And what are the differentiators and potential...
Ubiquiti Protect Video Surveillance Profile on Nov 07, 2018
Ubiquiti has now been in the video surveillance market for 7 years (see our first coverage back in 2011). In that time, the company's revenue has...
Directory Of Video Doorbells on Nov 06, 2018
Video doorbells are one of the fastest growing categories in video surveillance, especially among residences. The optimal placement of these...
Solar-Powered, Smart-Phone-Based Access Kit (VIZPin) Examined on Nov 02, 2018
Cloud-based access control company VIZPin is releasing a solar-powered and smart phone based access control system for gates and other remote...
IP Camera Installation Tool Shootout - Avigilon, Axis, Ideal, Hanwha, Triplett, Veracity on Oct 23, 2018
Setting up IP cameras has historically been challenging, with techs often precariously using a laptop on a ladder or lift. Some options for install...
IP Camera Installability Shootout - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Uniview, Vivotek on Oct 08, 2018
What are the best and worst cameras from an installation standpoint? Which manufacturers make it harder or easier to install their cameras? We...
Access Control Lock Guide on Sep 26, 2018
In this guide, we examine locks; critical elements of any security system and fundamental parts of every access control system. Two fundamental...
BluePoint Aims To Bring Life-Safety Mind-Set To Police Pull Stations on Sep 20, 2018
Fire alarm pull stations are commonplace but police ones are not. A self-funded startup, BluePoint Alert Solutions is aiming to make police pull...

Most Recent Industry Reports

The IP Camera Lock-In Trend: Meraki and Verkada on Jan 18, 2019
Open systems and interoperability have not only been big buzzwords over the past decade, but they have also become core features of video...
NYPD Refutes False SCMP Hikvision Story on Jan 18, 2019
The NYPD has refuted the SCMP Hikvision story, the Voice of America has reported. On January 11, 2018, the SCMP alleged that the NYPD was using...
Mobile Surveillance Trailers Guide on Jan 17, 2019
Putting cameras in a place for temporary surveillance where power and communications are not readily available can be complicated and expensive....
Exacq Favorability Results 2019 on Jan 17, 2019
Exacq favorability amongst integrators has declined sharply, in new IPVM statistics, compared to 2017 IPVM statistics for Exacq. Now, over 5 since...
Testing Bandwidth Vs. Low Light on Jan 16, 2019
Nighttime bandwidth spikes are a major concern in video surveillance. Many calculate bandwidth as a single 24/7 number, but bit rates vary...
Access Control Records Maintenance Guide on Jan 16, 2019
Weeding out old entries, turning off unused credentials, and updating who carries which credentials is as important as to maintaining security as...
UK Fines Security Firms For Illegal Direct Marketing on Jan 16, 2019
Two UK security firms have paid over $200,000 in fines for illegally making hundreds of thousands of calls to people registered on a government...
Access Control Cabling Tutorial on Jan 15, 2019
Access Control is only as reliable as its cables. While this aspect lacks the sexiness of other components, it remains a vital part of every...
Avigilon Favorability Results 2019 on Jan 15, 2019
Since IPVM's 2017 Avigilon favorability results, the company was acquired by Motorola and has shifted from being an aggressive startup to a more...
Gorilla Technology AI Provider, Raises $15 Million, Profiled on Jan 15, 2019
Gorilla Technology is a Taiwanese video analytics manufacturer that recently announced a $15 million investment from SBI Group, saying this...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact