Why 3VR Failed

Author: John Honovich, Published on Feb 16, 2018

3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit.

The reason they failed is simple. They bet on analytics. They lost.

The reason they bet on analytics is simple too. It is the same reason that Object Video bet on it in 2001 and Hikvision is betting on it in 2018. If you make it work, it has clear huge value.

But the more interesting question is: how did 3VR handle the problems? Obviously, they knew long before their firesale to Identiv that analytics had big problems. What did they try to do? Why did they not try other things? What lessons can be learned?

As someone who was on the inside during the early stages (2005 - 2007, first as Director of Systems Engineering, then as Director of Product Management), my goal is to share some perspective and ideas that might help illuminate the issues involved.

Positive - Lots of Smart People

One thing that certainly was not an issue was talent. 3VR had an abundance of talented people. The stereotype of Silicon Valley as a center for talent certainly proved true. One issue was a lack of experience in the surveillance industry but that was far less of a problem that not being able to deliver 3VR's core value proposition.

Problems Scaling - Demos vs Deployments

A key challenge 3VR experienced, and something still an issue today for many analytics companies, is a false sense of confidence from demos and small systems vs problems in larger production systems.

Like any analytic startup, 3VR started using their analytics in small areas - their own office with their own employees and people coming by for meetings (investors, prospects, etc.). With a small number of people to be analyzed and key engineering people to set things up precisely, 3VR worked pretty well inside 3VR's office. Likewise, in smaller beta and early deployments, especially with friendly customers, it 'mostly' worked.

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

The real problem came when 3VR's facial recognition started to be deployed in larger public sites (e.g., hotels or multiple bank branches). They had lots more people, lots more restrictions on where cameras could be placed, challenges in how people moved (large hallways, turns, etc.). This combined resulted in dramatically worse performance and false alerts / matches.

A major internal problem that resulted from this was a debate about why these results were so much worse. For example, many at HQ were understandably confused or skeptical about the problems in the field.

Pessimists vs Optimists

Obviously now, it is easy to look back and see the 'pessimists' (not surprisingly, myself included) were right. In fairness to the optimists, it was not as clear back then. It could have field errors and even if it was not that, the technology could have possibly improved so much in the next few years that the issues would be resolved.

Engineering Knew

On the other hand, after carefully analyzing various field results, key engineering people knew the problems faced and that eliminating them was far beyond any reasonable short to mid-term action. And, as the last decade showed, this is still a challenging problem today.

CEO Steve Russell - Great and Terrible

Enter CEO Steve Russell. Russell was simultaneously great and terrible.

He was incredible at getting people excited and convincing people things could be done. An incomparable pitchman, he combined self-confidence, humor, and intelligence in an extremely rare combination. One of the most amazing things he would do is talk about all levels of technology and technical details, but unlike engineers, he would say it confidently and without hesitation nor examining exceptions. This was terrific for a startup because when hard things came up (like facial recognition), he had great answers.

The problem was that he was terrible at understanding and accepting reality. While he sounded great talking about technology, after a while, you began to realize that Russell was engaging in wishful thinking, at best and science fiction, at worst. This was a huge problem, especially when 3VR faced real customer problems because Russell's reaction was to explain it away or offer some unrealistic but impressive sounding non-solution.

Engineering's Russell Rule

A great example of this was an informal 'Russell Rule' that the engineers had that no engineer should speak to Russell for more than 5 minutes. Russell was really good, in a bad way, at asking individual engineering specialists about specific problems, probing them to admit some way could be found. Then, the engineer, nervous not to offend the CEO, might agree, "well, maybe it's possible to do it that way", even though that was a theoretic possibility, not something 3VR could practically do in the foreseeable future.

Product Estimation Chaos

Related, there were comical product planning challenges with Russell. His estimates were consistently incredibly lower than others. Engineering leaders might estimate 6 weeks for a feature, I might say a month, Russell would say 3 days. He'd then pull out a white paper or article he read and explain how easy it was, while the exasperated engineering leaders tried in vain to explain why that was not realistic. This was, while funny at one level, a real problem for the company.

The Problem With Russell (and the Board's) Unrealistic Expectations

During the 2005 - 2007 time frame, as the company scaled up and expanded into more production deployments, 3VR undoubtedly faced significant performance problems. The debate was what to do.

Russel, and via Russell's great salesmanship, the board was sold on analytics and facial recognition as a game changer. The problem was 3VR could not deliver it in practice.

Selling Search vs Alerts

One common technique that 3VR tried to use was to emphasize selling search (i.e., 3VR is Google for video surveillance). So just like Google is not always perfect in giving a first result match and that does not matter, likewise 3VR would do the same. This, theoretically, would free 3VR from the much harder technical constraints of doing accurate facial surveillance alerts.

The sales problem was that video surveillance customers valued alerting far more than searching. Even if you steered customers to search, most would gravitate towards alerting and almost none would pay anywhere near the premium for better searching than they would for alerting.

Half Pivot To Recorder / Wells Fargo

3VR, to its credit, did at least half pivot to be a 'smart recorder'. While, at first, 3VR was facial surveillance with a few recorder features, 3VR did recognize that they needed to do both (facial + recorder) to win deals. And with a heroic effort by 3VR's then-junior but very talented salesperson Matt Vallone, plus engineering delivering dozens of fundamental recorder features, 3VR won Wells Fargo, 3VR's all-time largest customer and source of 30%+ of 3VR's total revenue.

Never Reached Product / Market Fit With Face

The problem, though, was that face recognition and other analytics never reached a point of performance where they could drive mass adoption. It increasingly became clear that 3VR was a solid recorder with some neat analytics that may or not be used, a no man's land in terms of growth.

No Pivot to Software Only / 'Just' A VMS

Two other important problems existed. These would not have been problems if face / analytics worked well, but without that, 3VR was left to fight against other recorders / VMS.

3VR was against software only for many years, only selling appliances, including the prime years where Genetec and Milestone were rapidly gaining ground. There were concerns about supporting customers. In retrospect, the decision clearly hurt the company.

Related, 3VR management was aware of the growth of the 'plain' non-analytic VMSes like Exacq, Genetec and Milestone. 3VR certainly had the money and the engineering talent to compete in that area. The problem was one of CEO and board expectations. To really focus and fight those VMS companies, 3VR would have to sacrifice development and focus on analytics which was something that they did not have the drive to do, simply it would have been effectively admitting failure. Of course, now, 3VR selling for $6.9 million in 2018, pivoting to be 'just' a VMS certainly would have delivered greater returns but the CEO and VCs would prefer to strike out going for a home run.

Leave to Do IPVM

I left in 2007 / 2008. While I did not know what I was going to do next, outside of programming on my own, that evolved into IPVM. And IPVM's critical coverage of marketing and analytics was definitely impacted and motivated by the challenges at 3VR.

The New CMO

The most notable 3VR exec to leave in that same time period was sales and marketing co-founder Tim Ross, who while a Silicon Valley native, was a realist and balance to Rusell.

3VR's next marketing leader Aisling MacRunnels was a stereotypical over-confident Silicon Valley exec. Despite being in the industry weeks, she lectured me about how analytics were so hot and how I (now running IPVM) did not understand the market. MacRunnels lasted 16 months at 3VR.

The New CEO

One of the strangest parts of 3VR was their hiring of Al Shipp, who had an impressive resume as an exec at Apple, stayed at 3VR for nearly a decade, overseeing its decline into this sorry ending. As an outsider, I am not sure what he was doing or thinking all that time.

Happy Ending For Russell

Lest anyone think everything here is negative for Russell, there is a happy ending for him. On the downside, he started another analytics company, Prism, that also has struggled. But, he was also friends or housemates with Garrett Camp who founded Uber. Russell put some money in early and, now, he has made far more on that than he is likely to make on his 2 analytic startups.

Lesson Learned

Given the constraints 3VR was operating under, specifically, the CEO and board's desire to go big and difficulty accepting the real challenges faced, it is not clear what could have been done.

However, trying as best as possible to understand the real problems a company face head-on, without excuses or wishful thinking, can result in properly changing direction to find a better outcome, even in a bad situation.

Comments (33)

Only IPVM PRO Members may comment. Login or Join.

AMA... I did not cover every topic as its long and complicated and was trying to condense it to some key interesting points. I am happy to expand on things as people would like to know. Interested to hear everyone's thoughts.

John you mentioned they failed because they bet on analytics. We use analytics for detection and we find it to be very reliable. Certainly in outdoor applications like car dealerships and construction sites it is far superior in both detection and cost effectiveness -to VM and other ways to secure. Once more we see it improving at an exponential pace, in this application.

Mark, they bet on analytics ~2005, and they bet primarily on face, ergo the issues. Things improve with time, though still not easy to do analytics and face rec.

Mark, btw, what manufacturer(s) are you using for analytics?

Thanks John that's what I thought for detection its the most cost effective but for other things I don't have much experience. We do video monitoring, remote concierge and gate control and we developed our own VMS with analytics from open source, works nicely in our central station software.

Matt V. was a stand up guy, and a pleasure to deal with. So am I reading this correct? There is a possibility, that customers that put down their cash for this system, were sold devices that had KNOWN failures internal to only 3VR? Hmmm, John where should the attorneys send the subpoena server to reach you?

This is maybe patient zero in my career for when I fully realized the accuracy of the saying “The marketing department is not connected to the reality department.”

were sold devices that had KNOWN failures internal to only 3VR? Hmmm, John where should the attorneys send the subpoena server to reach you?

Jeremy, we are not going to start with any softballs? :) Good questions.

The issue with 3VR was about the level of performance. Facial recognition worked to some degree. It was not faked or completely broken. The issue was how well it worked and how well buyers expected it to work.

And, even in 3VR, how good it worked was hotly debated. Even for people who acknowledged limitations in alerting, many argued passionately that search was where the value was.

One trend I noticed, even with some end users, is wishful thinking. Because it was 'cool', some people wanted it to work or overlooked the issues (which were displayed literally on screen in their offices).

The other more pratical thing that was done, that helped considerably, was eventually reducing prices, such that 3VR was often selling at the price of enterprise DVRs (at the time) while effectively giving away or charging very little for face / analytics.

More questions or comments, let me know.

In a previous comment I had stated that John's report was "on the money", but I do need to revise that slightly.

The facial recognition analytic worked as it should and as represented. As an end user, in 2008 we thoroughly tested the facial recognition analytic. I went to seven retail locations to conduct transactions as a customer. We then used transaction (point of sale) event data to find my image at one location. We then did a face match search across 30 locations and found facial event images of me at five of the seven locations. The two locations that did not capture my face as a facial event image had one analog color camera for every two tellers and the images of me did not meet the criteria for creating a "facial event" image.

We did not, however, end up using facial recognition. Here are some of the reasons:

Our deployment schedule was so aggressive that we did not certify that the installers enabled facial recognition, or when enabled that it was programmed correctly.

Generally, we did not upgrade cameras when we replaced all our VCRs with 3VR units. 3VR was very clear in telling us the criteria (camera positioning angles, pixels from "ear to ear", etc.).

Those on our investigations teams were very successful in using transaction data to search for and find suspects. It worked well. The success rate was high. The same success rate did not exist using facial recognition (FR), primarily due to inconsistencies in programming the recorders and because we did not replace and/or reposition our cameras where needed.

If we had installed to spec, I believe FR would have been successful and widely used. The more sophisticated and prolific fraudsters would open multiple accounts. In those cases our transaction data image searches using account data as a search field would not be successful but FR would have worked well.

...Al Shipp, who had an impressive resume as an exec at Apple, stayed at 3VR for nearly a decade, overseeing its decline into his sorry ending.

Did Russell abandon shipp?

Did Russell abandon shipp?

Bad pun, good question.

Keep in mind, Shipp joined in February 2009. While I had minimal communication with Shipp nor Russell at that time or later, my guess is the following.

Shipp was a salesperson and Russell was a great pitchman. And, at the beginning of 2009, from a sales perspective, things were great. Wells was just ramping up, Vallone had either just closed or was going to close another big chain. So Russell could genuinely focus on hockey stick growth and Shipp, as a salesperson from the outside, might have seen it as a great opportunity.

Related, Russell is the epitome of the George Costanza saying "it's not a lie if you believe it." I got the impression that Russell believed in what he was saying, no matter how unrealistic it might seem.

Did Russell abandon Shipp? Did he trick Shipp? I don't think any more than Russell tricked himself into believing.

What I'd really be curious to hear is why Shipp stayed for so long. Shipp may not have understood the problems in 2009 or 2010 but he had to figure out far before selling it off for pennies on the dollar in 2018.

What I'd really be curious to hear is why Shipp stayed for so long...

So when did Russell bail (npi), at the end or ?

Russell started Prism Skylabs (note: any company with labs in its name that is not running a lab is to be treated skeptically, e.g., BRS Labs) I believe in 2011, I know he was pitching it at Techcrunch Disrupt in September 2011 so he had left 3VR at least by then. Why and how he left I don't know. I was an outsider by then.

From the outside, my perception was that Prism was 3VR 2.0, i.e., Russell was trying to fix the problems he had at 3VR or at least re-focus to the new buzz, i.e., cloud / SaaS. Of course, again he bet on analytics and again that hit problems.

My experience with facial rec alerting is that it works* if:

1. You have an controlled / optimized FOV (near eye level like at customs, boarder crossings, etc.) -

2. You're hotlist database is local

3. You have enough CPU to run it (one quadcore per 1-2 cams)

* works meaning 80% + accuracy

Agree / Disagree or additional comments John?

The first point is key, that you have a controlled FOV, meaning only that the camera is straight on but the person is required to look at it. That radically improves performance.

But that's not what many or most people want with facial surveillance. They want a person walking in a wide store entrance with a hat on to be matched to a watch list. That's where you get into a whole lot of problems.

But that's not what many or most people want with facial surveillance. They want a person walking in a wide store entrance with a hat on to be matched to a watch list. That's where you get into a whole lot of problems.

Totally agree. It's an unrealistic expectation that some Face rec manufacturers still tell potential customers is achievable today.

Is IPVM ever going to do a face rec shootout?

Ever is a long time.

It does not make sense to do a shootout until multiple systems can pass preliminary / simple testing of low traffic and high accuracy. As yesterday's Hikvision results show, lots of systems have a lot of work to do.

A great opinion piece. A fun read!

I’m curious to hear what your opinion of how Gary Kallman handled his end of the business.

#4, it's a good question. The problem is that leading sales at 3VR was incredibly difficult given the fundamental product problems. Sometimes when you go through and fire a bunch of salespeople, you need to realize the salespeople are not the problem.

Take his predecessor - Mitch Ferguson - I am focusing on him because I was at 3VR the entire time he was there. He got shot. He struggled at 3VR. But he was clearly very talented and he went on, after 3VR, to have great success, with his next startup being sold to VMWare and the one after that IPOing.

As for Gary, he was coming when I was leaving so I don't know many details of his tenure but I can't imagine it was easy. Really nice guy and another person who has continued success post 3VR, now as President of First Alarm.

Gary’s responsibilities changed as new leaders were brought in.

Gary’s responsibilities changed

I don't know that era first hand. However, the impression I got was that 3VR management did not really know what to do. I blame that on the lack of product / market fit - 3VR was never a real focused VMS and it was never a formidable analytics platform, leaving it stuck in the middle. That's hard to manage around.

Who was financing Perkins and Kleiner?

You mean Kleiner Perkins? Like normally for VCs, they have various limited partners (LPs). And those LPs generally don't care who Kleiner is investing in as long as the returns are there.

Kleiner has had issues over the years, including poor relative returns and the Ellen Pao lawsuit / disaster.

I am not sure if any of that, even indirectly relates to 3VR. All VC firms make bad bets. For what it's worth, Kleiner's Ted Schlein, who led the 3VR investment is still at Kleiner, so surely he's had enough good exits to balance out 3VR.

3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit.

Does that include the $53 million that the CIA purportedly invested?

That's false. The CIA's VC firm, In-Q-Tel invested only a fraction of that.

Moreover, you would be more responsible to actually list where and how this claim is made.

The "$53 million" claim comes from 2013 Mattermark's List of 106 Startups Who Received Investment from the C.I.A. + Most Frequent In-Q-Tel Co-Investors, so it's not a primary source, it provides no citation nor reference to where this data comes from. Finally, it's under a table "Startups Who Received Investment from In-Q-Tel", column Total Funding, $53,000,000. I am not sure if Mattermark confused total 3VR funding with what they got just from In-Q-Tel.

Moreover, you would be more responsible to actually list where and how this claim is made.

I posted the link, same as you.

They provide a link to the CIA website which 404s, hence I said “purportedly”.

I posted the link, same as you.

No, not the same as me. I posted the name of the source, the title of the post and an explanation of how they came to the claim.

Moreover, the CIA link from the Mattermark article that 404s can be easily retrieved at Archive.org and has no reference to 3VR at all.

In all fairness to myself I was just asking a question, not making a statement.

Still, I provided the link to the “where” of the claim and questioned it by saying “purportedly”.

I provided all the information that I knew and could find at the time.

Found this:

So some part of $10,000,000. Is that in line with your recollection?

Yes, and that round was led by Kleiner, which means that In-Q-Tel was a minority of even that.

VC is a deal with the devil. At the pace the security industry moves I don’t believe VC is a good fit (there are exceptions) as they lack the patience to wait for their happy ending.

I would like to chime in on a few points. As 3VR's field engineer to most of the planet from 2007-2009, (essentially everywhere except west coast and Hawaii, and primary field contact at In-Q-Tel), I worked under Russell/Ross, Mitch Ferguson and Gary Kallman. I always thought that with a few additional (fairly simple) features, the platform should have been marketed as a VMS with neat search features (including the then unique ability to catalog faces). Marketing to financial institutions was always challenged by the false belief that the face-rec was a highly reliable anti-fraud tool. The intel-community folks who relied less on the machine and more on the human interraction had much more success with the product, even some substantial field results that can't be elaborated here. I thought the company would get over the hump when Gary Kallman came in. He was probably the best boss I ever had (except, of course, John Honovich). But as Gary came in, the primary product platform was still at an msrp of about $20k (eventually reduced to about $5k). I supported a lot of really great sales people (and managers), they seemed to turnover about once a year -- all essentially undermined by a failed marketing strategy that was forced on them. The product was always good. And my recollection of the investment from In-Q-Tel (at that time) was about $3 million, (and they put 3VR boxes in a number of federal agencies, including military and "3 letter" agencies for a variety of puposes -- many of which were successful to some extent).


I thought your report was right on the money. Matt Vallone was very good. But you didn't give yourself enough credit. You both contributed equally to landing the big account. And it didn't hurt that - as you said - the product worked and showed well. I would add that it was way high on the cool scale, and the UI was just plain fun.


Are you posting as Undisclosed #8? ;)

Related Reports

Free Trip To China - CCTV.Net / Univew on Mar 15, 2018
Pack your bags? 'Closer than you think'? Well, a non-stop flight from NYC to Shanghai is 15 hours plus another 100 miles to Hangzhou...
Network Addressing for Video Surveillance Guide on Mar 14, 2018
The goal of this guide is to explain addressing devices on IP networks, focusing on how IP cameras and recorders are used in those networks. For...
Panasonic Selling Off Security Camera Factory on Mar 14, 2018
Panasonic is OEMing cameras from Dahua, as IPVM testing confirmed in 2017. Now, Panasonic is selling their security camera factory, according to...
Favorite Camera Manufacturers 2018 on Mar 12, 2018
A number of major moves in integrator's favorite camera rankings for 2018: Two manufacturers make major moves up One major manufacturer moves...
Avigilon Breaks $400 Million Revenue, Quadruples Profit, Weakens Competitiveness on Mar 07, 2018
The last annual Avigilon financials are here. With the imminent Motorola acquisition completion, Avigilon will cease to be an independent publicly...
ADI W-Box Dropping Hikvision (Tested) on Mar 05, 2018
The next generation of ADI's W-Box (ADI's competition against their manufacturing partners) is here. And unlike the previous generation, which was...
ONVIF Usage Statistics 2018 on Mar 05, 2018
ONVIF has long 'won' the standards battle for video surveillance. But has the now 10-year-old ONVIF 'won' vs direct integrations? Undoubtedly,...
ADT: "Hate That Company" Loses Billion On Amazon Ring Deal on Mar 02, 2018
It has not been a good 2018 for ADT. And this week has been even worse. In the aftermath of Amazon acquiring Ring for a billion plus, ADT's...
Dynamic vs. Static IP Addresses Tutorial on Mar 01, 2018
While many cameras default to DHCP out of the box, that does not mean you should use it. This may seem basic for some, but those new to the...
Top Video Surveillance Service Call Problems (Statistics) on Feb 28, 2018
In our most recent statistics series, over 150 integrators told IPVM the most common problem their customers have with their video systems....

Most Recent Industry Reports

May 2018 Camera Course on Mar 16, 2018
Our next course starts on May 8th. Register now for the Spring 2018 Camera Course This is the only independent surveillance camera course, based...
ADT Hammered Again, Loses Another Billion In Market Cap on Mar 16, 2018
ADT's CEO told investors that, 'in baseball terms', ADT was batting 5 for 5. But investors told ADT's CEO, 'in baseball terms', that he was...
Camera Form Factor Guide on Mar 16, 2018
When selecting surveillance cameras, users may choose from a number of different form factors, each with its own unique strengths and weaknesses,...
Free Trip To China - CCTV.Net / Univew on Mar 15, 2018
Pack your bags? 'Closer than you think'? Well, a non-stop flight from NYC to Shanghai is 15 hours plus another 100 miles to Hangzhou...
Access Control - Restricted Keys Guide on Mar 15, 2018
Not all doors, even in larger facilities, can justify using electronic access control. And even for doors that do have electronic access control,...
Rack Mounting NVRs Tutorial on Mar 14, 2018
Rack mounting recorders is common in professional systems, but manufacturers are making it difficult, with simple design failures causing multiple...
Network Addressing for Video Surveillance Guide on Mar 14, 2018
The goal of this guide is to explain addressing devices on IP networks, focusing on how IP cameras and recorders are used in those networks. For...
Panasonic Selling Off Security Camera Factory on Mar 14, 2018
Panasonic is OEMing cameras from Dahua, as IPVM testing confirmed in 2017. Now, Panasonic is selling their security camera factory, according to...
Favorite Electrified Locks 2018 on Mar 14, 2018
Electronic lock manufacturing is dominated by 3 conglomerates (alphabetically) Allegion, Assa and Dormakaba holding numerous electronic lock...
Hikvision Chairman Joins China National Government (NPC) on Mar 13, 2018
Hikvision Chairman and Communist Party Secretary Chen Xongnian has joined the People's Republic of China's government - the National People's...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact