Why 3VR Failed

Published Feb 16, 2018 15:10 PM
PUBLIC - This article does not require an IPVM subscription. Feel free to share.

3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit. The reason they failed is simple. They bet on analytics. They lost.

IPVM Image

The reason they bet on analytics is simple too. It is the same reason that Object Video bet on it in 2001 and Hikvision is betting on it in 2018. If you make it work, it has clear huge value.

But the more interesting question is: how did 3VR handle the problems? Obviously, they knew long before their firesale to Identiv that analytics had big problems. What did they try to do? Why did they not try other things? What lessons can be learned?

As someone who was on the inside during the early stages (2005 - 2007, first as Director of Systems Engineering, then as Director of Product Management), my goal is to share perspectives and ideas that might help illuminate the issues involved.

Positive - Lots of Smart People

One thing that certainly was not an issue was talent. 3VR had an abundance of talented people. The stereotype of Silicon Valley as a center for talent certainly proved true. One issue was a lack of experience in the surveillance industry but that was far less of a problem than not being able to deliver 3VR's core value proposition.

Problems Scaling - Demos vs Deployments

A key challenge 3VR experienced, and something still an issue today for many analytics companies, is a false sense of confidence from demos and small systems vs problems in larger production systems.

Like any analytic startup, 3VR started using their analytics in small areas - their own office with their own employees and people coming by for meetings (investors, prospects, etc.). With a small number of people to be analyzed and key engineering people to set things up precisely, 3VR worked pretty well inside 3VR's office. Likewise, in smaller beta and early deployments, especially with friendly customers, it 'mostly' worked.

The real problem came when 3VR's facial recognition started to be deployed in larger public sites (e.g., hotels or multiple bank branches). They had lots more people, lots more restrictions on where cameras could be placed, challenges in how people moved (large hallways, turns, etc.). This combined resulted in dramatically worse performance and false alerts / matches.

A major internal problem that resulted from this was a debate about why these results were so much worse. For example, many at HQ were understandably confused or skeptical about the problems in the field.

Pessimists vs Optimists

Obviously now, it is easy to look back and see the 'pessimists' (not surprisingly, myself included) were right. In fairness to the optimists, it was not as clear back then. It could have been field errors and even if it was not that, the technology could have possibly improved so much in the next few years that the issues would be resolved.

Engineering Knew

On the other hand, after carefully analyzing various field results, key engineering people knew the problems faced and that eliminating them was far beyond any reasonable short to mid-term action. And, as the last decade showed, this is still a challenging problem today.

CEO Steve Russell - Great and Terrible

IPVM Image

Enter CEO Steve Russell - simultaneously great and terrible.

He was incredible at getting people excited and convincing people things could be done. An incomparable pitchman, he combined self-confidence, humor, and intelligence in an extremely rare combination. One of the most amazing things he would do is talk about all levels of technology and technical details, but unlike engineers, he would say it confidently and without hesitation nor examining exceptions. This was terrific for a startup because when hard things came up (like facial recognition), he had great answers.

The problem was that he was terrible at understanding and accepting reality. While he sounded great talking about technology, after a while, you began to realize that Russell was engaging in wishful thinking, at best and science fiction, at worst. This was a huge problem, especially when 3VR faced real customer problems because Russell's reaction was to explain it away or offer some unrealistic but impressive-sounding non-solution.

Engineering's Russell Rule

A great example of this was an informal 'Russell Rule' that the engineers had that no engineer should speak to Russell for more than 5 minutes. Russell was really good, in a bad way, at asking individual engineering specialists about specific problems, probing them to admit some way could be found. Then, the engineer, nervous not to offend the CEO, might agree, "well, maybe it's possible to do it that way", even though that was a theoretic possibility, not something 3VR could practically do in the foreseeable future.

Product Estimation Chaos

Related, there were comical product planning challenges with Russell. His estimates were consistently incredibly lower than others. Engineering leaders might estimate 6 weeks for a feature, I might say a month, Russell would say 3 days. He'd then pull out a white paper or article he read and explain how easy it was, while the exasperated engineering leaders tried in vain to explain why that was not realistic. This was, while funny at one level, a real problem for the company.

The Problem With Russell (and the Board's) Unrealistic Expectations

During the 2005 - 2007 time frame, as the company scaled up and expanded into more production deployments, 3VR undoubtedly faced significant performance problems. The debate was about what to do.

Russel, and via Russell's great salesmanship, the board was sold on analytics and facial recognition as a game changer. The problem was 3VR could not deliver it in practice.

Selling Search vs Alerts

One common technique that 3VR tried to use was to emphasize selling search (i.e., 3VR is Google for video surveillance). So just like Google is not always perfect in giving a first result match and that does not matter, likewise, 3VR would do the same. This, theoretically, would free 3VR from the much harder technical constraints of doing accurate facial surveillance alerts.

The sales problem was that video surveillance customers valued alerting far more than searching. Even if you steered customers to search, most would gravitate towards alerting and almost none would pay anywhere near the premium for better searching than they would for alerting.

Half Pivot To Recorder / Wells Fargo

3VR, to its credit, did at least half pivot to be a 'smart recorder'. While, at first, 3VR was facial surveillance with a few recorder features, 3VR did recognize that they needed to do both (facial + recorder) to win deals. And with a heroic effort by 3VR's then-junior but very talented salesperson Matt Vallone, plus engineering delivering dozens of fundamental recorder features, 3VR won Wells Fargo, 3VR's all-time largest customer and source of 30%+ of 3VR's total revenue.

Never Reached Product / Market Fit With Face

The problem, though, was that face recognition and other analytics never reached a point of performance where they could drive mass adoption. It increasingly became clear that 3VR was a solid recorder with some neat analytics that may or not be used, a no man's land in terms of growth.

No Pivot to Software Only / 'Just' A VMS

Two other important problems existed. These would not have been problems if face / analytics worked well, but without that, 3VR was left to fight against other recorders / VMS.

3VR was against software only for many years, only selling appliances, including the prime years when Genetec and Milestone were rapidly gaining ground. There were concerns about supporting customers. In retrospect, the decision clearly hurt the company.

Related, 3VR management was aware of the growth of the 'plain' non-analytic VMSes like Exacq, Genetec, and Milestone. 3VR certainly had the money and the engineering talent to compete in that area. The problem was one of CEO and board expectations. To really focus and fight those VMS companies, 3VR would have to sacrifice development and focus on analytics which was something that they did not have the drive to do, simply it would have been effectively admitting failure. Of course, now, 3VR selling for $6.9 million in 2018, pivoting to be 'just' a VMS certainly would have delivered greater returns but the CEO and VCs would prefer to strike out going for a home run.

Leave to Do IPVM

I left in 2007 / 2008. While I did not know what I was going to do next, outside of programming on my own, that evolved into IPVM. And IPVM's critical coverage of marketing and analytics was definitely impacted and motivated by the challenges at 3VR.

The New CMO

The most notable 3VR exec to leave in that same time period was sales and marketing co-founder Tim Ross, who while a Silicon Valley native, was a realist and balance to Rusell.

3VR's next marketing leader Aisling MacRunnels was a stereotypical over-confident Silicon Valley executive. Despite being in the industry for weeks, she lectured me about how analytics were so hot and how I (now running IPVM) did not understand the market. MacRunnels lasted 16 months at 3VR.

The New CEO

IPVM Image

One of the strangest parts of 3VR was their hiring of Al Shipp, who had an impressive resume as an exec at Apple, stayed at 3VR for nearly a decade, overseeing its decline into this sorry ending. As an outsider, I am not sure what he was doing or thinking all that time.

Happy Ending For Russell

Lest anyone thinks everything here is negative for Russell, there is a happy ending for him. On the downside, he started another analytics company, Prism, that also has struggled. But, he was also friends or housemates with Garrett Camp who founded Uber. Russell put some money in early and, now, he has made far more on that than he is likely to make on his 2 analytic startups.

Lesson Learned

Given the constraints 3VR was operating under, specifically, the CEO and board's desire to go big and difficulty accepting the real challenges faced, it is not clear what could have been done.

However, trying as best as possible to understand the real problems a company faces head-on, without excuses or wishful thinking, can result in properly changing direction to find a better outcome, even in a bad situation.

Comments (42)
JH
John Honovich
Feb 16, 2018
IPVM

AMA... I did not cover every topic as its long and complicated and was trying to condense it to some key interesting points. I am happy to expand on things as people would like to know. Interested to hear everyone's thoughts.

(7)
Avatar
Mark Palka
Feb 19, 2018
IPVMU Certified

John you mentioned they failed because they bet on analytics. We use analytics for detection and we find it to be very reliable. Certainly in outdoor applications like car dealerships and construction sites it is far superior in both detection and cost effectiveness -to VM and other ways to secure. Once more we see it improving at an exponential pace, in this application. 

(1)
JH
John Honovich
Feb 19, 2018
IPVM

Mark, they bet on analytics ~2005, and they bet primarily on face, ergo the issues. Things improve with time, though still not easy to do analytics and face rec.

Mark, btw, what manufacturer(s) are you using for analytics?

(1)
Avatar
Mark Palka
Feb 19, 2018
IPVMU Certified

Thanks John that's what I thought for detection its the most cost effective but for other things I don't have much experience. We do video monitoring, remote concierge and gate control and we developed our own VMS with analytics from open source, works nicely in our central station software.  

(1)
JE
Jeremy Ellis
Feb 16, 2018

Matt V. was a stand up guy, and a pleasure to deal with. So am I reading this correct? There is a possibility, that customers that put down their cash for this system, were sold devices that had KNOWN failures internal to only 3VR? Hmmm, John where should the attorneys send the subpoena server to reach you?

This is maybe patient zero in my career for when I fully realized the accuracy of the saying “The marketing department is not connected to the reality department.”

(2)
(2)
JH
John Honovich
Feb 16, 2018
IPVM

were sold devices that had KNOWN failures internal to only 3VR? Hmmm, John where should the attorneys send the subpoena server to reach you?

Jeremy, we are not going to start with any softballs? :) Good questions.

The issue with 3VR was about the level of performance. Facial recognition worked to some degree. It was not faked or completely broken. The issue was how well it worked and how well buyers expected it to work.

And, even in 3VR, how good it worked was hotly debated. Even for people who acknowledged limitations in alerting, many argued passionately that search was where the value was.

One trend I noticed, even with some end users, is wishful thinking. Because it was 'cool', some people wanted it to work or overlooked the issues (which were displayed literally on screen in their offices).

The other more pratical thing that was done, that helped considerably, was eventually reducing prices, such that 3VR was often selling at the price of enterprise DVRs (at the time) while effectively giving away or charging very little for face / analytics.

More questions or comments, let me know.

(1)
UE
Undisclosed End User #8
Mar 15, 2018

In a previous comment I had stated that John's report was "on the money", but I do need to revise that slightly.

The facial recognition analytic worked as it should and as represented.  As an end user, in 2008 we thoroughly tested the facial recognition analytic.  I went to seven retail locations to conduct transactions as a customer.  We then used transaction (point of sale) event data to find my image at one location.  We then did a face match search across 30 locations and found facial event images of me at five of the seven locations.  The two locations that did not capture my face as a facial event image had one analog color camera for every two tellers and the images of me did not meet the criteria for creating a "facial event" image. 

We did not, however, end up using facial recognition.  Here are some of the reasons:

Our deployment schedule was so aggressive that we did not certify that the installers enabled facial recognition, or when enabled that it was programmed correctly.

Generally, we did not upgrade cameras when we replaced all our VCRs with 3VR units.  3VR was very clear in telling us the criteria (camera positioning angles, pixels from "ear to ear", etc.).

Those on our investigations teams were very successful in using transaction data to search for and find suspects.  It worked well.  The success rate was high.  The same success rate did not exist using facial recognition (FR), primarily due to inconsistencies in programming the recorders and because we did not replace and/or reposition our cameras where needed.

If we had installed to spec, I believe FR would have been successful and widely used. The more sophisticated and prolific fraudsters would open multiple accounts.  In those cases our transaction data image searches using account data as a search field would not be successful but FR would have worked well.

(1)
(3)
U
Undisclosed #1
Feb 16, 2018
IPVMU Certified

...Al Shipp, who had an impressive resume as an exec at Apple, stayed at 3VR for nearly a decade, overseeing its decline into his sorry ending.

Did Russell abandon shipp?

(5)
JH
John Honovich
Feb 16, 2018
IPVM

Did Russell abandon shipp?

Bad pun, good question.

Keep in mind, Shipp joined in February 2009. While I had minimal communication with Shipp nor Russell at that time or later, my guess is the following.

Shipp was a salesperson and Russell was a great pitchman. And, at the beginning of 2009, from a sales perspective, things were great. Wells was just ramping up, Vallone had either just closed or was going to close another big chain. So Russell could genuinely focus on hockey stick growth and Shipp, as a salesperson from the outside, might have seen it as a great opportunity.

Related, Russell is the epitome of the George Costanza saying "it's not a lie if you believe it." I got the impression that Russell believed in what he was saying, no matter how unrealistic it might seem.

Did Russell abandon Shipp? Did he trick Shipp? I don't think any more than Russell tricked himself into believing.

What I'd really be curious to hear is why Shipp stayed for so long. Shipp may not have understood the problems in 2009 or 2010 but he had to figure out far before selling it off for pennies on the dollar in 2018.

 

(1)
(1)
U
Undisclosed #1
Feb 16, 2018
IPVMU Certified

What I'd really be curious to hear is why Shipp stayed for so long...

So when did Russell bail (npi), at the end or ?

JH
John Honovich
Feb 17, 2018
IPVM

Russell started Prism Skylabs (note: any company with labs in its name that is not running a lab is to be treated skeptically, e.g., BRS Labs) I believe in 2011, I know he was pitching it at Techcrunch Disrupt in September 2011 so he had left 3VR at least by then. Why and how he left I don't know. I was an outsider by then.

From the outside, my perception was that Prism was 3VR 2.0, i.e., Russell was trying to fix the problems he had at 3VR or at least re-focus to the new buzz, i.e., cloud / SaaS. Of course, again he bet on analytics and again that hit problems.

U
Undisclosed #9
May 27, 2020

any company with labs in its name that is not running a lab is to be treated skeptically

Nest was Nest Labs. FYI.

(1)
UM
Undisclosed Manufacturer #2
Feb 16, 2018

My experience with facial rec alerting is that it works* if:

1. You have an controlled / optimized FOV (near eye level like at customs, boarder crossings, etc.) - 

2. You're hotlist database is local

3. You have enough CPU to run it (one quadcore per 1-2 cams) 

* works meaning 80% + accuracy 

Agree / Disagree or additional comments John? 

(1)
(1)
JH
John Honovich
Feb 16, 2018
IPVM

The first point is key, that you have a controlled FOV, meaning only that the camera is straight on but the person is required to look at it. That radically improves performance.

But that's not what many or most people want with facial surveillance. They want a person walking in a wide store entrance with a hat on to be matched to a watch list. That's where you get into a whole lot of problems.

(3)
UM
Undisclosed Manufacturer #2
Feb 16, 2018

But that's not what many or most people want with facial surveillance. They want a person walking in a wide store entrance with a hat on to be matched to a watch list. That's where you get into a whole lot of problems.

Totally agree. It's an unrealistic expectation that some Face rec manufacturers still tell potential customers is achievable today. 

Is IPVM ever going to do a face rec shootout? 

JH
John Honovich
Feb 16, 2018
IPVM

Ever is a long time.

It does not make sense to do a shootout until multiple systems can pass preliminary / simple testing of low traffic and high accuracy. As yesterday's Hikvision results show, lots of systems have a lot of work to do.

UI
Undisclosed Integrator #3
Feb 16, 2018

A great opinion piece. A fun read!

UM
Undisclosed Manufacturer #4
Feb 16, 2018

I’m curious to hear what your opinion of how Gary Kallman handled his end of the business. 

JH
John Honovich
Feb 16, 2018
IPVM

#4, it's a good question. The problem is that leading sales at 3VR was incredibly difficult given the fundamental product problems. Sometimes when you go through and fire a bunch of salespeople, you need to realize the salespeople are not the problem.

Take his predecessor - Mitch Ferguson - I am focusing on him because I was at 3VR the entire time he was there. He got shot. He struggled at 3VR. But he was clearly very talented and he went on, after 3VR, to have great success, with his next startup being sold to VMWare and the one after that IPOing.

As for Gary, he was coming when I was leaving so I don't know many details of his tenure but I can't imagine it was easy. Really nice guy and another person who has continued success post 3VR, now as President of First Alarm.

(1)
UI
Undisclosed Integrator #7
Feb 19, 2018

Gary’s responsibilities changed as new leaders were brought in. 

JH
John Honovich
Feb 19, 2018
IPVM

Gary’s responsibilities changed

I don't know that era first hand. However, the impression I got was that 3VR management did not really know what to do. I blame that on the lack of product / market fit - 3VR was never a real focused VMS and it was never a formidable analytics platform, leaving it stuck in the middle. That's hard to manage around.

UM
Undisclosed Manufacturer #5
Feb 18, 2018

Who was financing Perkins and Kleiner?

JH
John Honovich
Feb 18, 2018
IPVM

You mean Kleiner Perkins? Like normally for VCs, they have various limited partners (LPs). And those LPs generally don't care who Kleiner is investing in as long as the returns are there.

Kleiner has had issues over the years, including poor relative returns and the Ellen Pao lawsuit / disaster.

I am not sure if any of that, even indirectly relates to 3VR. All VC firms make bad bets. For what it's worth, Kleiner's Ted Schlein, who led the 3VR investment is still at Kleiner, so surely he's had enough good exits to balance out 3VR.

U
Undisclosed #1
Feb 18, 2018
IPVMU Certified

3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit.

Does that include the $53 million that the CIA purportedly invested?

JH
John Honovich
Feb 19, 2018
IPVM

That's false. The CIA's VC firm, In-Q-Tel invested only a fraction of that.

Moreover, you would be more responsible to actually list where and how this claim is made.

The "$53 million" claim comes from 2013 Mattermark's List of 106 Startups Who Received Investment from the C.I.A. + Most Frequent In-Q-Tel Co-Investors, so it's not a primary source, it provides no citation nor reference to where this data comes from. Finally, it's under a table "Startups Who Received Investment from In-Q-Tel", column Total Funding, $53,000,000. I am not sure if Mattermark confused total 3VR funding with what they got just from In-Q-Tel.

 

(1)
U
Undisclosed #1
Feb 19, 2018
IPVMU Certified

Moreover, you would be more responsible to actually list where and how this claim is made.

I posted the link, same as you.

They provide a link to the CIA website which 404s, hence I said “purportedly”.

JH
John Honovich
Feb 19, 2018
IPVM

I posted the link, same as you.

No, not the same as me. I posted the name of the source, the title of the post and an explanation of how they came to the claim.

Moreover, the CIA link from the Mattermark article that 404s can be easily retrieved at Archive.org and has no reference to 3VR at all.

U
Undisclosed #1
Feb 19, 2018
IPVMU Certified

In all fairness to myself I was just asking a question, not making a statement.

Still, I provided the link to the “where” of the claim and questioned it by saying “purportedly”.  

I provided all the information that I knew and could find at the time.

 

 

U
Undisclosed #1
Feb 19, 2018
IPVMU Certified

Found this:

So some part of $10,000,000.  Is that in line with your recollection?

JH
John Honovich
Feb 19, 2018
IPVM

Yes, and that round was led by Kleiner, which means that In-Q-Tel was a minority of even that.

UM
Undisclosed Manufacturer #6
Feb 19, 2018

VC is a deal with the devil.  At the pace the security industry moves I don’t believe VC is a good fit (there are exceptions) as they lack the patience to wait for their happy ending.

SM
Stephen Matteo
Mar 12, 2018

I would like to chime in on a few points.  As 3VR's field engineer to most of the planet from 2007-2009, (essentially everywhere except west coast and Hawaii, and primary field contact at In-Q-Tel), I worked under Russell/Ross, Mitch Ferguson and Gary Kallman.  I always thought that with a few additional (fairly simple) features, the platform should have been marketed as a VMS with neat search features (including the then unique ability to catalog faces).  Marketing to financial institutions was always challenged by the false belief that the face-rec was a highly reliable anti-fraud tool.  The intel-community folks who relied less on the machine and more on the human interraction had much more success with the product, even some substantial field results that can't be elaborated here.  I thought the company would get over the hump when Gary Kallman came in. He was probably the best boss I ever had (except, of course, John Honovich).  But as Gary came in, the primary product platform was still at an msrp of about $20k (eventually reduced to about $5k).  I supported a lot of really great sales people (and managers), they seemed to turnover about once a year -- all essentially undermined by a failed marketing strategy that was forced on them.  The product was always good.  And my recollection of the investment from In-Q-Tel (at that time) was about $3 million, (and they put 3VR boxes in a number of federal agencies, including military and "3 letter" agencies for a variety of puposes -- many of which were successful to some extent).

      

(2)
(5)
UE
Undisclosed End User #8
Mar 15, 2018

John,

I thought your report was right on the money.  Matt Vallone was very good.  But you didn't give yourself enough credit.  You both contributed equally to landing the big account.  And it didn't hurt that - as you said - the product worked and showed well.  I would add that it was way high on the cool scale, and the UI was just plain fun.

(1)
UM
Undisclosed Manufacturer #4
Mar 15, 2018

John,

Are you posting as Undisclosed #8? ;)

 

(2)
Avatar
Kevin Mundy
Oct 07, 2022
Stanford University

put my kids through college with 3VR $$$.

(1)
JH
John Honovich
Oct 07, 2022
IPVM

BTW one thing I would add 4 years later is that analytics performance has significantly improved, driven by advances in neural networks and GPUs.

UI
Undisclosed Integrator #10
Oct 07, 2022

15-20 years is a long time in the technology world.

JH
John Honovich
Oct 07, 2022
IPVM

Sometimes yes, sometimes no. For example, the advances in the last 4 years were arguably more significant than the advances from say 2003 to 2018.

Avatar
Kevin Mundy
Oct 07, 2022
Stanford University

"Wait pipeline manager is blowing up, turn down the frame rate and resolution to keep it stable."

Things have come a long way since then, it was cool tech at the time. Like you said, sell search :-)

(1)
U
Undisclosed #11
Oct 07, 2022

Enter CEO Steve Russell - simultaneously great and terrible.

He was incredible at getting people excited and convincing people things could be done.

So was Elizabeth Holmes.

But seriously, 3VR seemed a little ahead of its time and should have waited for better cameras before it going to market. As with all analytics, even today, if the deployment specs are not adhered to, they won't work as expected.

(1)
Avatar
Kevin Mundy
Oct 07, 2022
Stanford University

Steve was really good at getting us all pumped up and excited for sure, it was a fun yet chaotic place to work. Soo many talented people in those earlier days.

Beer Friday's were fun!

UM
Undisclosed Manufacturer #13
Sep 07, 2023

Seems a good bit like the Oracle Network Computer (only difference being that 3VR didn't have an additional product line to keep it afloat after the failure).