32 NVRs Needed For ~1000 Avigilon Cameras?

1 NVR for every 33 cameras? 32 total NVRs needed for just over 1000?

Is this the right architecture?

A tweet's been going around with an Avigilon dealer showcasing such a project, see below:

A couple of things come to mind:

  • Only ~33 cameras per recorder is quite low. That low density likely increases cost.
  • With so many NVRs, it would likely be much less expensive, more efficient and easier to manage with external storage (SAN, NAS, etc.).

Specifically, the MSRP for those NVRs (with the lowest amount of storage - 3TB) is $5,160 and it gets much more expensive with more storage, excerpt:

And that, of course, does not include the cost of VMS licenses.

What approach would you take with this many cameras?


@unID LOL. Late nominee for post of the year.

Avigilon VMWare support statement:

"Avigilon confirms that we will support customers running License Plate Recognition and Avigilon Control Center Video Management on supported Operating Systems in a VMware virtual machine environment."

Here are some common approaches and rationales for managing large numbers of cameras:

  • More powerful servers managing 100+ cameras each. Rather than using NVR appliances, purchase COTS servers with multiple CPUs, higher amounts of RAM, etc. For example, the Avigilon NVRs come with a single CPU and 6GB RAM (see specifications). More powerful COTS servers can reduce total number of boxes needed and total cost.
  • Storage centralized in a SAN or NAS instead of storing inside of each NVR appliance (i.e., one central 'pool' of say 300TB vs 10TBs inside each NVR appliance). This is easier to manage and reduces inefficiencies as storage is shared amongst all servers. For example, you do not need to worry about any single NVR needing more storage or wasting storage on a particular NVR.
  • VMSes running on virtual machines often allow for more cameras to be recorded/managed on a single server (i.e., multiple VMS instances, one per VM) as well as facilitating transfer of that VM to a new machine (for service, fault tolerance or expansion).

With regards to Avigilon's NVR, they are specified to 32MB/s (i.e., 256Mb/s) throughput and/or 128 cameras. At 33 cameras per NVR, that implies an average bandwith consumption of 8 Mb/s, which is quite a lot even for full frame rate, multi megapixel continuously recorded video (unless it's JPEG2000 and then....)

Avigilon's employees and partners have gone berzerk on twitter, mostly with ad hominin attacks (calling us 'such a dick', etc.) but have not offered any specific facts or technical details that would explain the decision.

1. Avigilon will not record to NAS

2. 256 Mb/sec standard advise by lots of VMS companies

3. 8 Mb/sec about right for H.264 specially at night time

I personally never put more then 24-30 cams per server ( 2MP at 30 frames )

Virtual Machines choice is up to customer not up to me

I offer they decide

there is a potential for this having been a install specified by an engineer... in that case spec may have called for avigilon nvrs vs. the cots servers you mention... sometimes you can substitute with an equal or greater product... without much for details I am just speculating...

The most telling part was I wasn't even recommending running the OS on a VM, I was personally thinking about the storage aspect. That didnt seem to dawn on anyone...
Really the number of cameras per server is irrelevant without bandwidth. Also, many times customers plan on expansion so completely loading servers to the max may not be preferred. Without complete details this is absurd speculation and completely fruitless albeit entertaining

Great tactic. So Avigilon won't share any details so no one is allowed to discuss or question it.

What is very useful is people understanding the most common architectures used for large scale video deployments, why they are used and how this goes against them.

Here are some more plausible theories for the deal / approach:

  • This was a negotiated project between an Avigilon dealer and the customer. The dealer, wanting to maximize revenue and profits, advocated an all Avigilon NVR approach.
  • This was influenced by an Avigilon RSM / A&E rep who persuaded the engineer to go with all Avigilon NVRs.

The scenario where an engineer out of the blue specified an all Avigilon NVR deployment with no interaction or encouragement from Avigilon is far less likely.

Pushing appliances over software only + COTS hardware is a common tactic of any vendor who offers both (I've seen the same with Exacq).

In this deal, selling appliances instead of sw only + COTS generated Avigilon a few hundred thousand more in revenue, the dealer tens of thousands more in profits and the sales reps thousands more in commissions.

The financial motive to go appliances is clear. The rationale for going this architecture over the more scalable, lower cost, more common approach is not.

"Avigilon will not record to NAS"

Are you saying Avigilon will not record directly to NASes (which is completely understandable) or that Avigilon's VMS can not send video to a NAS or SAN for storage? If it's the latter, it's a huge competitive flaw.

"256 Mb/sec standard advise by lots of VMS companies"

This is a function of server specs, not simply application. That's my point of using a higher end COTS server vs the Avigilon NVR. Can Avigilon literally not handle anything more than 256Mb/s even with more computing resources that what the NVR provides? If so, that's a significant competitive flaw.

"8 Mb/sec about right for H.264 specially at night time"

That's presuming all cameras are recording continuously, at 30fps and are not using a VBR cap and do not have integrated IR (which would lower bandwidth) or are in lighted areas. This may very well be the case (though it's doubtful) and I welcome Avigilon to share facts here.

Great tactic John deleting my post

1.What this have to do with Avigilon ?

It was just a Picture posted by integrator on Tweeter

who may or may not wish to share details

That HIS choice

By the way I did not see anywhere on your website wishing members Merry Christmas

kinda mean

Your entire "post" was just "+1"

I deleted it because we have a voting function that allows you to do just that. It's redundant and a waste of space.

Merry Christmas!

I'm not sure how to interpret the tense they have chosen. Is it akin to Amazon's "we will" for drone deliveries?

Without knowing the application and customers concerns, objectives and requirements it impossible to say whether this is a good design for this application.

We would not by default design or engineer a system like this but we have been asked by some of our customers to provide solutions like this and believe it or not there is some very valid logic/reasoning for requesting solutions like this

Considerations

Why choose only Avigilon for servers/storage, cams and software? Customer wants complete turn key solution from one vendor "one throat to choke / no finger pointing / one call for service". You may find it weird but our largest customers are the ones that request this the most for standardization, efficiency and support.

Why not consolidate servers storage? Customer wants to minimize / limit the exposure / affected devices in any single piece of equipment failure (including servers and storage).

Why so few cameras per server? Customer has cameras set to high frame rates and bandwidth reducing the number of cameras per server or System is designed for worst case bitrates Or customer plans on doing failover or camera additions in the future and has budget now for servers... or bla bla bla.

These are some possible reasons behind a design like this - but...

Again, without knowing the application and customers concerns, objectives and requirements it impossible to say whether this is a good design for this application.

Thanks for laying out some potential rationales.

Feedback:

"Customer wants complete turn key solution from one vendor"

Totally believable though in this case I suspect they are paying a significant premium vs a COTS server + SAN approach.

"Customer wants to minimize / limit the exposure / affected devices in any single piece of equipment failure"

Why not then only record 16 channels per appliance or 1 channel per appliance? This way the exposure from a single piece of equipment failing is even less. Without some stronger case that higher cameras counts significantly increases risks, it's hard to understand why 33 cameras per recorder is the best combination.

"Why so few cameras per server?"

Even if frame rates and bitrates or high, the point remains that a COTS server with higher specifications than the Avigilon NVR most certainly would record many more cameras, even at high fps or bitrates.

Since Avigilon will most likely never disclose any details of what's involved, everyone is left to making educated guesses but this discussion does highlight key factors in the decision making process.

The one point that absolutely remains is that Avigilon dealers make more money selling Avigilon NVRs than COTS servers and storage, incenting them to favor the former.

Why does Avigilon have to share anything with you? Because you said so?

What do you expect when you pervoke people into getting what you want?

Customer has 2 options

COTS machines with little warranty

Avigilon servers with 4 hour warranty

Customer gets to choose and is paying extra for the warranty

Avigilon uses very little CPU and memery so building COTS with faster CPU and more memory will not "most certainly would record many more cameras"

Without knowing customers needs/wants your just making stuff up.

X

Customer has 2 options - or maybe the integrator only gave them these two options.

COTS machines with little warranty; or Avigilon servers with 4 hour warranty - this 4 hour 'warranty' is not a warranty at all... it is a service agreement. The same dealer could give the same 4 hour service agreement on either hardware choice if they chose to. Of course if they only offer '2 choices' it is easier to push the Avigilon boxes vs COTS machines that this same dealer will not offer the same service contract on.

Customer gets to choose and is paying extra for the warranty service contract - If the dealer only offers one choice with a 4 hour service contract, then they are, in effect, making the choice for the customer.

Avigilon uses very little CPU/memory so building COTS with faster CPU and more memory will not 'most certainly would record many more cameras' - If Avigilon uses so little CPU/memory, then why only 33 cams/box?

Without knowing the customers needs/wants you're just making stuff up - So does this customer need to spend more money than required, or do they just want to?

Alex, Avigilon NVRs are just OEM'ed Dell boxes. You must know this, right? If not, here's the Dell / Avigilon OEM marketing brochure.

So any claimed warranty or service benefits are Dell features that one could get from buying from Dell directly.

The issue here is the architecture for handling 1000+ cameras. This customer almost certainly would save money, energy and space by choosing a more appropriate higher end Dell server and network based storage. They'd still get the same 'quality' and support by buying from Dell directly.

Avigilon's NVR appliances are perfectly fine and appropriate for smaller cameras counts but deploying more than 30 for 1000 cameras is most likely a flawed technical approach.

Alex,

It's in Avigilon's interests to address this because IPVM is very well read among people that Avigilon wants to influence. You've seen the end users, investors, partners, competitors, potential new hires, etc. that read IPVM.

If Avigilon has facts that shows why this is a sensible overall solution, than they will gain from proving their case. As it is, this discussion is becoming yet another valuable resource for Avigilon competitors to game plan against future Avigilon proposals similar to this one.

In the absence of Avigilon speaking on its own behalf, you have become their spokesman. Not good.... for them.

John

"most likely a flawed technical approach"

In your opinion without knowing ANYTHING about the project and customers needs/wants.

You cleary have a vendetta against Avigilon

Alex,

We know three very critical pieces of information about the project: (1) the camera count, (2) the appliance count and (3) that they are all in one site. Given those core details, most likely this is a flawed technical approach.

If there are some secondary details that would justify it, feel free to share. We've already debunked your COTS 'little warranty' fallacy. What seems to be left is the technical flaws you are suggesting about Avigilon - can't record to network storage, can't handle more than 256Mb/s per box. Right?

John what you know is enough to speculate - that's it.

I will say this - we have servers from Avigilon running over 100 megapixel IP cameras on a single server and we have some running over 180 analog cameras on a single server. However we have some running 30 or fewer cameras as well. We have applications where we go with servers straight from dell and create a few VMs on them and utilize centralized SAN for storage. We also have applications with customer provided storage and servers. Crap - we actually have Avigilon running on $400 pcs from Walmart with 40 analog cams and a few IP no problem. The fact here is all of these applications we utilized the appropriate servers and storage based on the specific requirements, concerns and budgets for each customer. Every one of these customers are happy as well.

I see no reason why Avigilon or the integrator should try to explain the application on this site simply because you decided to post an article. This does not give Avigilons competitors any type of advantage either. Trying to use this speculative, incomplete ponderance of an article to gain a competitive advantage would be ignorant.

I know as well as you by now that you do these articles for fun probably when you are bored or have nothing else to write about - to see what responses you will get. This holds no real value to any of your users (other than entertainment) because it is completely speculative. It's a "bait" article - good bait though trying to get a company or your users to defend a product because of your negative speculation...

Nice to see that somebody else have balls :)

On another hand John does not realise that with "negative" talks about Avigilon

He actually help Avigilon big time

last year lots of new companies became partners with Avigilon

Thanks John

Happy new Year and plz keep talking about Avigilon

"The fact here is all of these applications we utilized the appropriate servers and storage based on the specific requirements, concerns and budgets for each customer. Every one of these customers are happy as well."

In the real world, all deployments are not accurate and optimized designs based on what's best for the customer. Many mistakes are made, often because of incentives to sell more product (e.g., NVR appliances instead of COTS servers).

This discussion examines a very interesting and important aspect of video surveillance - what are the design options and tradeoffs for larger scale, single site systems? The case for COTS servers / VMs / SANs over NVR appliances is quite strong.

Finally, if you want to speculate on motives, it's ironic that you continue to comment only to defend your business partner. Perhaps you want to stick up for Genetec and Panasonic as we recently questioned the value of one of their large deals...

IPVM will continue to question, examine and call out projects, technologies and products for many companies. Merry Christmas.

"Real world"...seriosly ?

When did you last time get up from your chair went out sell ,design,install and service system ?

I would strongly suggest do it in new year

then you and us will be in the same boat

Happy new year

Alex, our organization does extensive testing, research and surveys of the industry around the globe. We also routinely offer private consulting and designs to end user members at no additional cost.

If you do not think we know the real world, then I would suggest you not waste your money on an IPVM membership.

Btw, perhaps for the New Year, you can improve your English writing skills to at least a 13 year old's level, especially now that you are Avigilon's de facto spokesperson....

Getting personal as usual

by the way English is my 4th language

and also try to figure out why Avigilon partners so loyal to Avigilon

It could be your tenth, the issue is that this site is in English. Your writing skills are so poor, it is routinely difficult to understand your point. Moreover, criticism in such incredibly botched English like "When did you last time get up from your chair went out sell ,design,install and service system ?" carries far less weight.

The same thing that makes Avigilon partners so loyal to them (making money on restricted sales) is what makes their feedback on technical topics so botched - it's biased, flawed and in your case, typically unintelligible.

As this is a solely speculative discussion and you are speculating this is a bad design I'm simply stating there could be good (relative) reason behind this design. I do not know the installer or the application. I'm just providing some insight into possibilities other than the only negative possibilities laid out by John. As "proper" design is relative AND dependent on so many factors that are not provided in this discussion we have absolutely no clue who is closer in the ppossibilities laid out.

Ultimately, if the customer is happy with the end solutuon and within budget everything else is a moot point.

Lots of Avigilon people are saying John is merely speculating when saying that 32 NVRs running 1000 cameras is not the most optimum architecture for this deployment.

Please provide ANY reasoning that makes sense for utilizing such an architecture in THIS particular install.

Thank you.

He actually help Avigilon big time

Alex don't blow his cover, he's actually heading up Avigilon's RPPR (Reverse Psychology Public Relations) effort;

Didn't you get the memo?

Yes, I did :)

and u right

negative ads (conversations) works very well

If I remember correctly it's call "boomerang effect" :)

I would like to talk to you on ph

Is it possible ?

Everything with THIS is speculation because we do not have any details other than camera count and the number of servers.

It's easy to give a preferred design method based on camera count alone but literally impossible to know wether it is acceptable or preferred without the rest of the solution details / requirements and the customer preference and input.

Good integrators design systems based on each customers requirements and preferences. We have customers that have us do things we would never recommend or design but that's the way they want it - so we do it.

Without knowing the complete project requirements and the customers preference and input noone can speak to what's right for THIS application - period

Your question reveals both your inability to understand previous comments and your ignorance when it comes to system design and customer input.

There are 2 potential architectures being considered:

  • Dozens of NVR appliances (OEMed from Dell)
  • A Few COTS servers + network based storage

You and Alex have done a great job of making it seem that these two approaches are equally valid for a 1000 camera, single site project. This is false.

For this scenario, regardless of the unknowns, the benefits of a few COTS servers + network based storage over NVR appliances are myriad and significant - much lower product cost, less space needed, lower power consumption, greater scalability, less storage wasted, etc.

Please refute this specifically. The only potential viable argument I have heard so far is Alex's implication that Avigilon's technical flaws are what prevents this (COTS servers + network based storage) from being pursued.

I am sure insulting all of your non english members is a great tactic for increasing participation

Anything anyone says about Avigilon you twist around to your benefit so what is the point?

Would be happy to have converation in my native language but I would say the same about you "it's biased, flawed and in your case, typically unintelligible."

X

So you know John, with only the information provided, we would lead with a design as you suggest utilizing more robust servers and a centralized SAN. So we agree that superficially this appears to be a better architecture for an application like this and one even we would suggest and present initially. But just because this would be our initial design does not mean that our customer would agree or accept the design. Therein lies our disagreement and what I think is one of your weakest mental attributes: how to deal with customers / end users in the real world - they matter even if they disagree and ultimately we are providing a solution for them so we have to listen to them and build our design around our knowledge and experience along with our customers specific requirements, concerns, preferences and input. Sometimes the solutions we implement are radically different than what we have designed or preferred because of our customers standards, requirements and preferences. The fact is we do not know any other details about this implementation and this customer so even though we have our design preferences that we agree on does not in any way allow us to assert with absolute authority that it would be best for this application and customer.

I would like to wish everyone a Merry Christmas! Hope you all are having a great day and have the opportunity to spend valuable quality time with all those that are important in your life.

If this was the first reply from the Avigilon camp, I imagine this whole string would never have occured.

Happy Holidays everyone! :)

Undisclosed Seasons Greetings to you also!

Send me your P.O. box#, so we can exchange Christmas Cards!

U

John, Sean: I think it would be helpful to members if you could identify the material facts in this case.

On first reading all I see is

  1. A rack with what appear to be 13 double height nvrs
  2. A negative statement regarding how infrequently one sees 32 nvrs

Regressing thru the twitter thread I see a statement about 1000 cameras by someone different than the one making the 32 nvr statement. I can't really say I saw much 'showcasing'.

Are there other tweets/conversations that are relevant?

Thx Marty!

Perhaps John could (or has already) reach(ed) out to JP for clarification. Maybe even offer him a temp login for a roundtable discussion??

On the bright side even Ryno_Video admits "It's not often you get to see 32 Avigilon NVRs on a site.", so such low density configurations for whatever reasons may be not very common.

P.S. If the 32 nvrs are in a single rack we're talking about one of these puppies:

64U rack

Now that's a rack!

FWIW only three of the NVRs seem to be on...

I have to agree with some of the undisclosed about "Real World". Sometimes providing the right solution isn't what is going to get you the job, because unfortunately someone else will give the customer what they want, even if it's wrong.

It could be that the customer demanded 32 Avigilon NVRs while the dealer desperately tried to persuade them to get COTS servers and a SAN. Or it could be the dealer pushed what would make them the most money - all Avigilon NVRs.

The latter is a far more common scenario.

Chris,

My understanding is that Avigilon employees, per their management, are not allowed to speak with or to IPVM.

You're stuck with Alex K...

John

I am sure John's antagonistic and condescending tweets will win them over.

X

As both a Dell direct partner and an Avigilon dealer, I can tell you that there is not all that much difference in cost to me between the two servers. While I read most if these comments I did not read them all (this seems to be a hot topic), did anyone consider that maybe the dealer decided to utilize Avigilon servers for the sheer fact that Avigilon would have not have a leg to stand on if there were performance issues surrounding the servers? I can't tell you how many times i have installed servers that meet or exceed the "Recommended" specs only to have a manufacturer question them right out of the box if we were discussing performance issues. The next thing they question is the network itself, which is usually dropped once I ask them which if the 3 Cisco CCIE's in my office they want to discuss that with.

I have my first medium scale Avigilon install within the next 30 days (365 streams), I will let you know how it goes. I hope all I am hearing here about Avigilon is just a bunch if noise, but if it is not, the engagement will be short lived.

Some VMS systems can only handle video databases of a certain size. The larger the database gets, the slower playback and retrieval time gets. And then some just have a maximum limit, like any SQL or Oracle database. You can get more powerful machines, maybe, but some databases only scale to certain sizes, so you have to spread out overall size across multiple machines. (And more powerful servers and SANS sometimes cost almost as much as multiple smaller servers.) 33 cameras maybe not seem like a lot of cameras that would make a sizable database, but if the client requires a very long retention time (60 days... 90 days.... 180 days), that's a large databse for just a few cameras.

You're stuck with Alex K...

Stuck? Hardly, my Pappy always said 'if life gives you lemons...'.

Two thing stand out:

We can talk to Alex

No one in their right mind could possibly mistake Alex for IPVM

Therefore supplemental to the 'Christmas Day Massacre', I propose the first ever

New Year's Eve Security Seance!

Whereby the celebrated Mr. K attempts to "channel" his partners and coax answers and specs from the Great Beyond!

Done right this could get national attention ('60 Minutes' shouldn't have a problem with it). I can imagine the 'teaser spots' already:

Moderator: Alex where are you; what do you see?

Mr K: I am presently in what appears to be a small room or utility closet, alone, yet surrounded by a steel structure with perhaps two dozen multi-colored orbs, blinking and whirring in a most peculiar fashion.

Moderator: Can you get a good look inside the structure?

Mr K: i am trying but there's a lock.... Wait, Wait! A large man with a larger document is coming towards me at great speed...

Moderator: Alex!!

----

So, Alex K will you make contact?

[Remove]

Chris

would you like me to post my ph number for you ?

in case it will get delete it

u can join Avigilon Dealers group on LinkedIn

This is a good point about database size - we used to run into this regularly on older Vigil DVRs, where if an MSSQL database exceeded the memory allocated to it, searches would bog down substantially (fixes included running database repair/compression or a full rebuild to clean up old entries, or adding RAM and adjusting the memory allocation accordingly). Fortunately with newer versions, they've gone to using multiple smaller databases and allowing Vigil to adjust the memory allocation automatically to avoid this issue, but depending on the system, I could see it continuing to be an issue.

You're just lucky it really is, #622. ;)

Thank you for acknowledging the fundamental point here - that for the core details we know, "robust servers and a centralized SAN" is a better approach. This is why it's a topic worth discussing and an important point for surveillance professionals to understand when considering large scale systems.

You presume that the customer 'disagreed' with what we both agree on and that the customer pushed the Avigilon dealer to go with all Avigilon NVR appliances.

This is ironic since you simultaneously keep harping how we don't know enough but evidently you know enough to confidently assert that it was the customer's choice to go with all Avigilon NVRs rather the dealer who advocated it. Do you see the catch-22 you are in?

It could be either one (the customer demanded or the dealer pushed for it) but given who benefits financially in selling more Avigilon NVRs vs COTS servers + SANs, it is far more likely that the dealer was the one who advocated this approach.

John, your pen (keyboard) is mightier than your mind. I presume absolutely nothing - you should take off your negative filter and read my posts again. I have never said I thought one thing over another as you have - I simply provided an alternate to your speculation. Again, I know it's tough for you to understand but speculation is just that and holds no true value to your users other than entertainment. Actually I know you know it and you like to stir things up - good for your site traffic but bad for actual content value.

What's truly great is Avigilon refuses to speak to you because of your tactics and you just can't stand it - it's hilarious. You think you should be able to force a manufacturer to defend themselves because you choose to speculate over a few twitter feeds and then you twist it around with your own twitter feeds... Classy John - and professional because we know security professionals make their decisions based on the speculation and rantings of a "non-biased" guy like yourself.

IPVTMZ is here. At this rate your average user will change from end users and security / other professionals to teenage girls - umm and me because I'm a sucker for this crap obviously.

Your whole counterargument above is premised on the customer disagreeing with (both of our) recommendations for "robust servers and a centralized SAN" approach. Read your comment again and you will see. All I am saying is that of those two 'speculations', mine is far more likely as it follows the financial incentives. You can feel free to disagree but let's leave it at that.

That said, you are a rabid Avigilon fanboi who only ever objects to IPVM content when it is critical about Avigilon. When we criticize other companies in the same way, like the recent Genetec and Panasonic project, nothing...

Keep on popping away, as you acknowledge, you are a sucker for defending Avigilon.