Why 3VR Failed

Author: John Honovich, Published on Feb 16, 2018

3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit.

The reason they failed is simple. They bet on analytics. They lost.

The reason they bet on analytics is simple too. It is the same reason that Object Video bet on it in 2001 and Hikvision is betting on it in 2018. If you make it work, it has clear huge value.

But the more interesting question is: how did 3VR handle the problems? Obviously, they knew long before their firesale to Identiv that analytics had big problems. What did they try to do? Why did they not try other things? What lessons can be learned?

As someone who was on the inside during the early stages (2005 - 2007, first as Director of Systems Engineering, then as Director of Product Management), my goal is to share some perspective and ideas that might help illuminate the issues involved.

Positive - Lots of Smart People

One thing that certainly was not an issue was talent. 3VR had an abundance of talented people. The stereotype of Silicon Valley as a center for talent certainly proved true. One issue was a lack of experience in the surveillance industry but that was far less of a problem that not being able to deliver 3VR's core value proposition.

Problems Scaling - Demos vs Deployments

A key challenge 3VR experienced, and something still an issue today for many analytics companies, is a false sense of confidence from demos and small systems vs problems in larger production systems.

Like any analytic startup, 3VR started using their analytics in small areas - their own office with their own employees and people coming by for meetings (investors, prospects, etc.). With a small number of people to be analyzed and key engineering people to set things up precisely, 3VR worked pretty well inside 3VR's office. Likewise, in smaller beta and early deployments, especially with friendly customers, it 'mostly' worked.

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

The real problem came when 3VR's facial recognition started to be deployed in larger public sites (e.g., hotels or multiple bank branches). They had lots more people, lots more restrictions on where cameras could be placed, challenges in how people moved (large hallways, turns, etc.). This combined resulted in dramatically worse performance and false alerts / matches.

A major internal problem that resulted from this was a debate about why these results were so much worse. For example, many at HQ were understandably confused or skeptical about the problems in the field.

Pessimists vs Optimists

Obviously now, it is easy to look back and see the 'pessimists' (not surprisingly, myself included) were right. In fairness to the optimists, it was not as clear back then. It could have field errors and even if it was not that, the technology could have possibly improved so much in the next few years that the issues would be resolved.

Engineering Knew

On the other hand, after carefully analyzing various field results, key engineering people knew the problems faced and that eliminating them was far beyond any reasonable short to mid-term action. And, as the last decade showed, this is still a challenging problem today.

CEO Steve Russell - Great and Terrible

Enter CEO Steve Russell. Russell was simultaneously great and terrible.

He was incredible at getting people excited and convincing people things could be done. An incomparable pitchman, he combined self-confidence, humor, and intelligence in an extremely rare combination. One of the most amazing things he would do is talk about all levels of technology and technical details, but unlike engineers, he would say it confidently and without hesitation nor examining exceptions. This was terrific for a startup because when hard things came up (like facial recognition), he had great answers.

The problem was that he was terrible at understanding and accepting reality. While he sounded great talking about technology, after a while, you began to realize that Russell was engaging in wishful thinking, at best and science fiction, at worst. This was a huge problem, especially when 3VR faced real customer problems because Russell's reaction was to explain it away or offer some unrealistic but impressive sounding non-solution.

Engineering's Russell Rule

A great example of this was an informal 'Russell Rule' that the engineers had that no engineer should speak to Russell for more than 5 minutes. Russell was really good, in a bad way, at asking individual engineering specialists about specific problems, probing them to admit some way could be found. Then, the engineer, nervous not to offend the CEO, might agree, "well, maybe it's possible to do it that way", even though that was a theoretic possibility, not something 3VR could practically do in the foreseeable future.

Product Estimation Chaos

Related, there were comical product planning challenges with Russell. His estimates were consistently incredibly lower than others. Engineering leaders might estimate 6 weeks for a feature, I might say a month, Russell would say 3 days. He'd then pull out a white paper or article he read and explain how easy it was, while the exasperated engineering leaders tried in vain to explain why that was not realistic. This was, while funny at one level, a real problem for the company.

The Problem With Russell (and the Board's) Unrealistic Expectations

During the 2005 - 2007 time frame, as the company scaled up and expanded into more production deployments, 3VR undoubtedly faced significant performance problems. The debate was what to do.

Russel, and via Russell's great salesmanship, the board was sold on analytics and facial recognition as a game changer. The problem was 3VR could not deliver it in practice.

Selling Search vs Alerts

One common technique that 3VR tried to use was to emphasize selling search (i.e., 3VR is Google for video surveillance). So just like Google is not always perfect in giving a first result match and that does not matter, likewise 3VR would do the same. This, theoretically, would free 3VR from the much harder technical constraints of doing accurate facial surveillance alerts.

The sales problem was that video surveillance customers valued alerting far more than searching. Even if you steered customers to search, most would gravitate towards alerting and almost none would pay anywhere near the premium for better searching than they would for alerting.

Half Pivot To Recorder / Wells Fargo

3VR, to its credit, did at least half pivot to be a 'smart recorder'. While, at first, 3VR was facial surveillance with a few recorder features, 3VR did recognize that they needed to do both (facial + recorder) to win deals. And with a heroic effort by 3VR's then-junior but very talented salesperson Matt Vallone, plus engineering delivering dozens of fundamental recorder features, 3VR won Wells Fargo, 3VR's all-time largest customer and source of 30%+ of 3VR's total revenue.

Never Reached Product / Market Fit With Face

The problem, though, was that face recognition and other analytics never reached a point of performance where they could drive mass adoption. It increasingly became clear that 3VR was a solid recorder with some neat analytics that may or not be used, a no man's land in terms of growth.

No Pivot to Software Only / 'Just' A VMS

Two other important problems existed. These would not have been problems if face / analytics worked well, but without that, 3VR was left to fight against other recorders / VMS.

3VR was against software only for many years, only selling appliances, including the prime years where Genetec and Milestone were rapidly gaining ground. There were concerns about supporting customers. In retrospect, the decision clearly hurt the company.

Related, 3VR management was aware of the growth of the 'plain' non-analytic VMSes like Exacq, Genetec and Milestone. 3VR certainly had the money and the engineering talent to compete in that area. The problem was one of CEO and board expectations. To really focus and fight those VMS companies, 3VR would have to sacrifice development and focus on analytics which was something that they did not have the drive to do, simply it would have been effectively admitting failure. Of course, now, 3VR selling for $6.9 million in 2018, pivoting to be 'just' a VMS certainly would have delivered greater returns but the CEO and VCs would prefer to strike out going for a home run.

Leave to Do IPVM

I left in 2007 / 2008. While I did not know what I was going to do next, outside of programming on my own, that evolved into IPVM. And IPVM's critical coverage of marketing and analytics was definitely impacted and motivated by the challenges at 3VR.

The New CMO

The most notable 3VR exec to leave in that same time period was sales and marketing co-founder Tim Ross, who while a Silicon Valley native, was a realist and balance to Rusell.

3VR's next marketing leader Aisling MacRunnels was a stereotypical over-confident Silicon Valley exec. Despite being in the industry weeks, she lectured me about how analytics were so hot and how I (now running IPVM) did not understand the market. MacRunnels lasted 16 months at 3VR.

The New CEO

One of the strangest parts of 3VR was their hiring of Al Shipp, who had an impressive resume as an exec at Apple, stayed at 3VR for nearly a decade, overseeing its decline into this sorry ending. As an outsider, I am not sure what he was doing or thinking all that time.

Happy Ending For Russell

Lest anyone think everything here is negative for Russell, there is a happy ending for him. On the downside, he started another analytics company, Prism, that also has struggled. But, he was also friends or housemates with Garrett Camp who founded Uber. Russell put some money in early and, now, he has made far more on that than he is likely to make on his 2 analytic startups.

Lesson Learned

Given the constraints 3VR was operating under, specifically, the CEO and board's desire to go big and difficulty accepting the real challenges faced, it is not clear what could have been done.

However, trying as best as possible to understand the real problems a company face head-on, without excuses or wishful thinking, can result in properly changing direction to find a better outcome, even in a bad situation.

2 reports cite this report:

2018 Mid-Year Surveillance Industry Guide on Jun 28, 2018
2018 has been an explosive year for the video surveillance industry, with the industry becoming a global political issue, with the expansion of...
How To Become an 'Expert' In An Hour on Sep 02, 2016
If you want to be perceived as an expert in your market, speak at every event possible... the math is simple: when someone stands in front of an...

Comments (33)

Only IPVM PRO Members may comment. Login or Join.

AMA... I did not cover every topic as its long and complicated and was trying to condense it to some key interesting points. I am happy to expand on things as people would like to know. Interested to hear everyone's thoughts.

John you mentioned they failed because they bet on analytics. We use analytics for detection and we find it to be very reliable. Certainly in outdoor applications like car dealerships and construction sites it is far superior in both detection and cost effectiveness -to VM and other ways to secure. Once more we see it improving at an exponential pace, in this application.

Mark, they bet on analytics ~2005, and they bet primarily on face, ergo the issues. Things improve with time, though still not easy to do analytics and face rec.

Mark, btw, what manufacturer(s) are you using for analytics?

Thanks John that's what I thought for detection its the most cost effective but for other things I don't have much experience. We do video monitoring, remote concierge and gate control and we developed our own VMS with analytics from open source, works nicely in our central station software.

Matt V. was a stand up guy, and a pleasure to deal with. So am I reading this correct? There is a possibility, that customers that put down their cash for this system, were sold devices that had KNOWN failures internal to only 3VR? Hmmm, John where should the attorneys send the subpoena server to reach you?

This is maybe patient zero in my career for when I fully realized the accuracy of the saying “The marketing department is not connected to the reality department.”

were sold devices that had KNOWN failures internal to only 3VR? Hmmm, John where should the attorneys send the subpoena server to reach you?

Jeremy, we are not going to start with any softballs? :) Good questions.

The issue with 3VR was about the level of performance. Facial recognition worked to some degree. It was not faked or completely broken. The issue was how well it worked and how well buyers expected it to work.

And, even in 3VR, how good it worked was hotly debated. Even for people who acknowledged limitations in alerting, many argued passionately that search was where the value was.

One trend I noticed, even with some end users, is wishful thinking. Because it was 'cool', some people wanted it to work or overlooked the issues (which were displayed literally on screen in their offices).

The other more pratical thing that was done, that helped considerably, was eventually reducing prices, such that 3VR was often selling at the price of enterprise DVRs (at the time) while effectively giving away or charging very little for face / analytics.

More questions or comments, let me know.

In a previous comment I had stated that John's report was "on the money", but I do need to revise that slightly.

The facial recognition analytic worked as it should and as represented. As an end user, in 2008 we thoroughly tested the facial recognition analytic. I went to seven retail locations to conduct transactions as a customer. We then used transaction (point of sale) event data to find my image at one location. We then did a face match search across 30 locations and found facial event images of me at five of the seven locations. The two locations that did not capture my face as a facial event image had one analog color camera for every two tellers and the images of me did not meet the criteria for creating a "facial event" image.

We did not, however, end up using facial recognition. Here are some of the reasons:

Our deployment schedule was so aggressive that we did not certify that the installers enabled facial recognition, or when enabled that it was programmed correctly.

Generally, we did not upgrade cameras when we replaced all our VCRs with 3VR units. 3VR was very clear in telling us the criteria (camera positioning angles, pixels from "ear to ear", etc.).

Those on our investigations teams were very successful in using transaction data to search for and find suspects. It worked well. The success rate was high. The same success rate did not exist using facial recognition (FR), primarily due to inconsistencies in programming the recorders and because we did not replace and/or reposition our cameras where needed.

If we had installed to spec, I believe FR would have been successful and widely used. The more sophisticated and prolific fraudsters would open multiple accounts. In those cases our transaction data image searches using account data as a search field would not be successful but FR would have worked well.

...Al Shipp, who had an impressive resume as an exec at Apple, stayed at 3VR for nearly a decade, overseeing its decline into his sorry ending.

Did Russell abandon shipp?

Did Russell abandon shipp?

Bad pun, good question.

Keep in mind, Shipp joined in February 2009. While I had minimal communication with Shipp nor Russell at that time or later, my guess is the following.

Shipp was a salesperson and Russell was a great pitchman. And, at the beginning of 2009, from a sales perspective, things were great. Wells was just ramping up, Vallone had either just closed or was going to close another big chain. So Russell could genuinely focus on hockey stick growth and Shipp, as a salesperson from the outside, might have seen it as a great opportunity.

Related, Russell is the epitome of the George Costanza saying "it's not a lie if you believe it." I got the impression that Russell believed in what he was saying, no matter how unrealistic it might seem.

Did Russell abandon Shipp? Did he trick Shipp? I don't think any more than Russell tricked himself into believing.

What I'd really be curious to hear is why Shipp stayed for so long. Shipp may not have understood the problems in 2009 or 2010 but he had to figure out far before selling it off for pennies on the dollar in 2018.

What I'd really be curious to hear is why Shipp stayed for so long...

So when did Russell bail (npi), at the end or ?

Russell started Prism Skylabs (note: any company with labs in its name that is not running a lab is to be treated skeptically, e.g., BRS Labs) I believe in 2011, I know he was pitching it at Techcrunch Disrupt in September 2011 so he had left 3VR at least by then. Why and how he left I don't know. I was an outsider by then.

From the outside, my perception was that Prism was 3VR 2.0, i.e., Russell was trying to fix the problems he had at 3VR or at least re-focus to the new buzz, i.e., cloud / SaaS. Of course, again he bet on analytics and again that hit problems.

My experience with facial rec alerting is that it works* if:

1. You have an controlled / optimized FOV (near eye level like at customs, boarder crossings, etc.) -

2. You're hotlist database is local

3. You have enough CPU to run it (one quadcore per 1-2 cams)

* works meaning 80% + accuracy

Agree / Disagree or additional comments John?

The first point is key, that you have a controlled FOV, meaning only that the camera is straight on but the person is required to look at it. That radically improves performance.

But that's not what many or most people want with facial surveillance. They want a person walking in a wide store entrance with a hat on to be matched to a watch list. That's where you get into a whole lot of problems.

But that's not what many or most people want with facial surveillance. They want a person walking in a wide store entrance with a hat on to be matched to a watch list. That's where you get into a whole lot of problems.

Totally agree. It's an unrealistic expectation that some Face rec manufacturers still tell potential customers is achievable today.

Is IPVM ever going to do a face rec shootout?

Ever is a long time.

It does not make sense to do a shootout until multiple systems can pass preliminary / simple testing of low traffic and high accuracy. As yesterday's Hikvision results show, lots of systems have a lot of work to do.

A great opinion piece. A fun read!

I’m curious to hear what your opinion of how Gary Kallman handled his end of the business.

#4, it's a good question. The problem is that leading sales at 3VR was incredibly difficult given the fundamental product problems. Sometimes when you go through and fire a bunch of salespeople, you need to realize the salespeople are not the problem.

Take his predecessor - Mitch Ferguson - I am focusing on him because I was at 3VR the entire time he was there. He got shot. He struggled at 3VR. But he was clearly very talented and he went on, after 3VR, to have great success, with his next startup being sold to VMWare and the one after that IPOing.

As for Gary, he was coming when I was leaving so I don't know many details of his tenure but I can't imagine it was easy. Really nice guy and another person who has continued success post 3VR, now as President of First Alarm.

Gary’s responsibilities changed as new leaders were brought in.

Gary’s responsibilities changed

I don't know that era first hand. However, the impression I got was that 3VR management did not really know what to do. I blame that on the lack of product / market fit - 3VR was never a real focused VMS and it was never a formidable analytics platform, leaving it stuck in the middle. That's hard to manage around.

Who was financing Perkins and Kleiner?

You mean Kleiner Perkins? Like normally for VCs, they have various limited partners (LPs). And those LPs generally don't care who Kleiner is investing in as long as the returns are there.

Kleiner has had issues over the years, including poor relative returns and the Ellen Pao lawsuit / disaster.

I am not sure if any of that, even indirectly relates to 3VR. All VC firms make bad bets. For what it's worth, Kleiner's Ted Schlein, who led the 3VR investment is still at Kleiner, so surely he's had enough good exits to balance out 3VR.

3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit.

Does that include the $53 million that the CIA purportedly invested?

That's false. The CIA's VC firm, In-Q-Tel invested only a fraction of that.

Moreover, you would be more responsible to actually list where and how this claim is made.

The "$53 million" claim comes from 2013 Mattermark's List of 106 Startups Who Received Investment from the C.I.A. + Most Frequent In-Q-Tel Co-Investors, so it's not a primary source, it provides no citation nor reference to where this data comes from. Finally, it's under a table "Startups Who Received Investment from In-Q-Tel", column Total Funding, $53,000,000. I am not sure if Mattermark confused total 3VR funding with what they got just from In-Q-Tel.

Moreover, you would be more responsible to actually list where and how this claim is made.

I posted the link, same as you.

They provide a link to the CIA website which 404s, hence I said “purportedly”.

I posted the link, same as you.

No, not the same as me. I posted the name of the source, the title of the post and an explanation of how they came to the claim.

Moreover, the CIA link from the Mattermark article that 404s can be easily retrieved at Archive.org and has no reference to 3VR at all.

In all fairness to myself I was just asking a question, not making a statement.

Still, I provided the link to the “where” of the claim and questioned it by saying “purportedly”.

I provided all the information that I knew and could find at the time.

Found this:

So some part of $10,000,000. Is that in line with your recollection?

Yes, and that round was led by Kleiner, which means that In-Q-Tel was a minority of even that.

VC is a deal with the devil. At the pace the security industry moves I don’t believe VC is a good fit (there are exceptions) as they lack the patience to wait for their happy ending.

I would like to chime in on a few points. As 3VR's field engineer to most of the planet from 2007-2009, (essentially everywhere except west coast and Hawaii, and primary field contact at In-Q-Tel), I worked under Russell/Ross, Mitch Ferguson and Gary Kallman. I always thought that with a few additional (fairly simple) features, the platform should have been marketed as a VMS with neat search features (including the then unique ability to catalog faces). Marketing to financial institutions was always challenged by the false belief that the face-rec was a highly reliable anti-fraud tool. The intel-community folks who relied less on the machine and more on the human interraction had much more success with the product, even some substantial field results that can't be elaborated here. I thought the company would get over the hump when Gary Kallman came in. He was probably the best boss I ever had (except, of course, John Honovich). But as Gary came in, the primary product platform was still at an msrp of about $20k (eventually reduced to about $5k). I supported a lot of really great sales people (and managers), they seemed to turnover about once a year -- all essentially undermined by a failed marketing strategy that was forced on them. The product was always good. And my recollection of the investment from In-Q-Tel (at that time) was about $3 million, (and they put 3VR boxes in a number of federal agencies, including military and "3 letter" agencies for a variety of puposes -- many of which were successful to some extent).

John,

I thought your report was right on the money. Matt Vallone was very good. But you didn't give yourself enough credit. You both contributed equally to landing the big account. And it didn't hurt that - as you said - the product worked and showed well. I would add that it was way high on the cool scale, and the UI was just plain fun.

John,

Are you posting as Undisclosed #8? ;)

Related Reports

Hanwha Dual Imager Dome Camera Tested (PNM-7000VD) on Oct 18, 2018
Hanwha has introduced their first dual-imager model, the PNM-7000VD, a twin 1080p model featuring independently positionable sensors and a snap-in...
Knightscope Winning Investors, Struggling With Growth on Oct 16, 2018
While Knightscope's new financials show the company only winning 11 new customers in the past 12 months, the company continues to win new...
Integrator Laptop Guide on Oct 16, 2018
This 18-page guide provides guidance and statistics about integrator laptop use. 150 integrators explained to IPVM in detail about their laptops,...
Huawei Admits AI "Bubble" on Oct 16, 2018
A fascinating article from the Chinese government's Global Times: Huawei’s AI ambition to reshape industries. While the Global Times talks about...
Dahua Face Recognition Camera Tested on Oct 15, 2018
Dahua has been one of the industry's most vocal proponents of the value that AI creates: As part of this, Dahua has released a facial...
Amazon Touts Home Security Market Disruption on Oct 15, 2018
Amazon is coming for ADT and all of home security. Indeed, Amazon is advertising this as, in their own words, calling home security a: Inside...
Mysterious Patent Troll 'Secure Cam' Targets Industry, Sues Hanwha, Hikvison, JCI, Panasonic, More on Oct 11, 2018
A company named "Secure Cam," who is actively hiding their ownership, has acquired a slew of video patents and is systematically suing video...
Security System Health Monitoring Usage Statistics 2018 on Oct 09, 2018
How well and quickly do integrators know if devices are offline or broken? New IPVM statistics show that typically no health monitoring is...
IACP 2018 Police Show Final Report on Oct 08, 2018
IPVM went to Orlando to cover the 2018 IACP conference, the country's largest police show (about as big as ASIS), examining the 700+...
Evidence Of Hikvision's Involvement With Xinjiang IJOP And Re-Education Camps on Oct 02, 2018
IPVM reveals as-yet unreported details about Hikvision’s activities in Xinjiang - a region of China where massive human rights abuses are being...

Most Recent Industry Reports

Startup SafePass Profile on Oct 19, 2018
A major problem with visitor management is that the systems mostly require adhesive printed paper labels and paper logs, creating waste and an...
China Is Not A Security Megatrend, Says SIA on Oct 19, 2018
The US Security Industry Association has released its 10 "Security Megatrends" for 2019. SIA declares that these megatrends, such as "Advanced...
Hanwha Dual Imager Dome Camera Tested (PNM-7000VD) on Oct 18, 2018
Hanwha has introduced their first dual-imager model, the PNM-7000VD, a twin 1080p model featuring independently positionable sensors and a snap-in...
Camera Height / Blind Spot Added to IPVM Camera Calculator on Oct 18, 2018
IPVM has added camera height and blind spot estimation to the Camera Calculator. This is especially helpful for those who need to mount cameras up...
Axis Strong US Growth, Flat EMEA - Q3 2018 Financials on Oct 18, 2018
This spring, Axis had its best financials in many years (see Axis Strong Q2 2018 Results). However, over the summer, Axis had many products sold...
Best Alternatives to Banned Dahua and Hikvision on Oct 17, 2018
With the US government ban and a growing number of users banning Dahua and Hikvision, one key question is what to use for low cost? While Dahua and...
Video Quality / Compression Tutorial on Oct 17, 2018
While CODECs, like H.264, H.265, and MJPEG, get a lot of attention, a camera's 'quality' or compression setting has a big impact on overall...
Knightscope Winning Investors, Struggling With Growth on Oct 16, 2018
While Knightscope's new financials show the company only winning 11 new customers in the past 12 months, the company continues to win new...
Integrator Laptop Guide on Oct 16, 2018
This 18-page guide provides guidance and statistics about integrator laptop use. 150 integrators explained to IPVM in detail about their laptops,...
Huawei Admits AI "Bubble" on Oct 16, 2018
A fascinating article from the Chinese government's Global Times: Huawei’s AI ambition to reshape industries. While the Global Times talks about...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact