Subscriber Discussion

Milestone Recording Server - Milestone Recommended DELL Vs DIY Server

LC
Lew Cody
Feb 04, 2018
IPVMU Certified

Hi Everyone. Could I solicit opinions and input relating to the Recording Server hardware requirements for a 30-IPcam (4mp or 5mp) Milestone VMS system. This is a 'simple' application in a retail environment with no corporate or high end requirements.

Milestone are recommending their 'best practice' DELL Intel Xeon E5-2620 @ $US6500. We believe this is a "CMA" overkill.

We believe we can do the same job with an Intel i7 (5960 or thereabouts) with a 500Gb SSD (drive or PCI slot) as the base recording server storage and the traditional 16TB archive storage. Milestone is (understandably) reluctant to endorse this.

We have previously built our own Recording Servers with an i7 CPU and a Samsung 500Gb SSD (to grab the realtime video streams) which has run now for around 24 months at about 12% CPU utilization @ 12fps. OK - if we wind the cams to 30fps, then the CPU starts to top out at 100%, but who needs 30fps anyway. We are happy with 12fps and 12% CPU utilization.

Is Milestone pushing their agenda with DELL sales?

What is you experience in installing Milestone VMS on 'DIY' servers?

Are there justifiable reasons why one should buckle under and install the DELL?

What Server spec. would you consider adequate if you had to build one (or have one built) for 30x5mp IP cams @ 12fps?

In some ways one feels a bit 'intimidated' by the sales and tech talk from Milestone and how it is BEST to stick with what they recommend.

We just want to be sure that our thinking is correct.

We know there may be a number of variables to consider, but we just want a ball-park opinion here.

Thanks in advance!

Lewis

 

 

 

(1)
RS
Robert Shih
Feb 04, 2018
Independent

Not so quick question: why are we sticking with the 5th generation architecture?

Even better question: why is what Milestone recommending even more dated with fewer cores?

On another note, is Milestone effectively making use of that many cores in the first place?

Are there any hyperthreading optimizations? I've actually wondered about this with many high level VMS solutions.

A LOT of these systems are choosing very dated hardware (Dahua is no exception).

Have you benched this against the latest i7-8700K that now comes with 6 cores (12 threads)?

If more cores/threads are actually better unequivocally, why not AMD's new Threadripper?

In fact, how much R&D is being done to see how well new hardware improves upon VMS performance? And pointing this in the other direction: how often do VMS programmers take advantage of new hardware improvements and optimize their engines?

(7)
(2)
Avatar
Walter Holm
Jun 13, 2018
IPVMU Certified

Xeons are recommended for server loads all the time for the following reasons: L3 cache sizes, ECC memory support and they are built to be under constant load.  You can get more "free" out of an i7 but that doesn't mean it will last or perform like the Xeon either.

#CPUs, I feel it is still early of multi-threaded video operations but it can be done, this is more of a developer question to ask really but if you are applying a lot of reason to process (scaling video, large number of client support, etc.) then yes (this can be independent of multi-threading).

It is hard for a manufacturer to constantly come up with best practice hardware because I think it depends on the environment.  A small business might not be able to tolerate the high cost of a server like above and maybe building a DIY out a NUC is better.

Intel vs AMD, maybe they are complier optimized for Intel?

I believe the biggest help in video processing is using the GPU offloading, not number of CPUs.

YV
Yoeri Valkenborghs
Feb 04, 2018

And There’s also a big difference between the normal and the +versions.

it seems most of the time the calculator is a bit outdated.

perhaps you can look at the Promise Technologies systems, they are optimized for use with Milestone and to this with lower end processors.

 

LC
Lew Cody
Feb 04, 2018
IPVMU Certified

Exactly - those are reasonable observations.

I would imagine that any recommended solution by a company such as Milestone would lag behind the curve of cutting edge technology. 

1. Because DELL et all still have to brand and test their own architecture before releasing it

2. They have to clear old stock before making way for the new

3. They are stuck on a slow conveyor belt and the new stuff will eventually come along

In the meantime, here we are stuck in the same rut if we don't find the best way to implement outside the so-called 'best practice' concept.

For us it is a question of what is considered a sufficiently powerful CPU architecture.

 

LC
Lew Cody
Feb 04, 2018
IPVMU Certified

Oh just BTW - when we implemented with a 500Gb Samsung SSD, it is rated at 6Gb/s throughput. Milestone technical told us (confidentially), that this configuration would knock the s**t out of their DELL which didn't have close to that throughput. Go figure!

Referring to the short term storage drive that receives realtime data

(1)
(2)
Avatar
Marc Pichaud
Feb 04, 2018

The point is that I 've seen that 150 mbit/s (30 x 2MP & 5MP and couples of 12MP, 12 fps GOP1) recording streams  on a E-SATA  7,2 K RPM (and please no purple 5400 RPM )  pre streaming Server in randomized recording is not working, you get quickly overflow, not on the server (I7, overdimensionned < 14% CPU) but on the RAID controler and Disk heads ....It could generate tons of Network errors between Server and Archiver ! (TCP noAck, resending..) and simply NO RECORDS at all or missing damages files in the database

With Milestone, you need to read the small caracters recommandations  , they are not an option but mandatory:  RAID10 Hard controler (no soft) AND SAS disks (12Gb/s) for high Writting perf (don't care the reading as it will be sequentially archived later on on RAID5 E-SATA) which requires much lower writting speeds and good reading skills

(2)
(2)
LC
Lew Cody
Feb 04, 2018
IPVMU Certified

Hi Marc, yes, agreed - capturing the pre-recording streams is the core component.

With a PCI SSD one can get a theoretical 32Gb/s which exceeds a RAID spec..or am I wrong? Even a traditional SSD gives one 7Gb/s.

However, the question remains...will a DELL system give better performance and why?

Are there any valid technical reasons why a standard hi spec CPU with SSD pre-recording will not do the same job? What is the justification for settling for a Milestone recommended server?

Maybe someone has done it both ways, with a Milestone recommended server and with a traditional i7 machine? Would be interesting to know the assessment of performance.

Avatar
Josh Hendricks
Feb 04, 2018
Milestone Systems

You're in Oz right Lew? I lived and worked with Tas & Angelo and the team for four years and FWIW I can tell you there is no ulterior motive behind recommending Dell over something else. There's probably some measure of CYA and recommending what you're familiar with/had great success with though.

I'm guessing you're comparing the Dell recommendation to a "white box"? That is likely the concern the team has. Will your setup work? Probably! But Milestone's end customers have been burned before by partners installing lower quality equipment.

With Dell (or HP or IBM etc) you're getting...

Components that are more likely to be well tested together and be known to work well

Probably better warranty/service from the manufacturer compared to white box systems

ILO makes remote service so much more effective

Probably more likely to get driver updates for various components.

There's a higher cost to go Dell, but then you make up some amount of that by not having to spend time designing and building servers, managing warranty for individual components etc.

If the customer ever decides to switch integrators for any reason, it's better for the customer to have familiar hardware with warranty/service than to have a proprietary box.

Many integrators don't have the technical knowledge to diagnose hardware or performance issues, so Milestone Support is often the one forced to diagnose this and convince the integrator which is sometimes a challenging conversation and expensive for the integrator. We don't want to spend our time doing this and we don't want you to either.

 

That doesn't mean a white box, or just a similar HP server isn't going to work or be supported by Milestone. But our recommendations are going to be made for equipment we are familiar with, and have some reasonable amount of overhead. I encourage you to have a construction conversation with our team about the reasons behind the recommendation. I'm guessing you'll get some version of this. In the end, we just want you to have a successful install and for the customer be happy with their Milestone system.

(8)
(4)
Avatar
Michael Dawkins
Feb 21, 2018
IPVMU Certified

Hey Lew, did you ever think to use DW spectrum. I totally believe is building your own server, once you've done your homework on the hardware specification then your server would outshine the dell and do it at a far lower cost. I am also suggesting using DW spectrum as the VMS

 

(2)
(4)
(1)
UM
Undisclosed Manufacturer #1
Feb 05, 2018

I've seen a situation where Milestone was constantly crashing on newer, high end HP servers. Was it Milestones fault, or HP's? You can use all the high end, popular components you want, but that doesn't mean they will work properly together all the time, or even for all the same software.

I think it really is safer if you at least go with a system builder who has supposedly tested VMS software with their systems and will back it up, like either Seneca Data or BCDVideo to name a few if you are not happy with the Dell recommendation.

(5)
Avatar
Walter Holm
Jun 13, 2018
IPVMU Certified

Vendor quality has waned at times.  HP is no exception.  I am not with their recent high end servers.  I have never said that about Dell...yet.  Before HP, IBM used to be great and they suffered a quality build problem.  HP has been having major firmware issues lately.  Dell, I have not seen that happen...yet.

Avatar
Mike Dotson
Feb 05, 2018
Formerly of Seneca • IPVMU Certified

UI1 is correct in that we (Seneca) do performance testing with many different VMSs in house and have learned what the nuances are of each one from a performance perspective.   A vanilla box will not run as well as one that is properly tuned for the task at hand.  We do this testing for Servers and Clients as each needs different resources. This is part of what marketing likes to call Value Add.

Indeed an SSD as a target for the new data will be very fast, but in the long term they will wear out very fast.  Pay attention to the endurance rating of the drives you select.

A 5MP-12FPS stream can be in the 5 to 9Mbits range depending on the scene and activity it sees.   This puts the camera ingest at around 250Mbits.

Be aware of the different design points of the CPUs used and the jobs you are asking them to do.   The 'I-series' is made for regular office workloads and gamers in a limited hourly span, whereas the Xeons are made to run 24/7-365.   The i7-8700K is an awesome CPU but you will wear it out if you are pushing it hard.

The current Xprotect 2017R3 code will take advantage of Intel HD GFX for server side motion detection very nicely.   The SmartClient function also uses this.  Their whitepapers for the server side are actually accurate.

The Xeon E3-12x5 ones have this feature.  The i7-59xx you mentioned does NOT.

Not mentioned was the use of RAID storage and how the streams are viewed.

You do NOT want to be viewing these 4/5MP streams locally on the recorder server as it will interfere with writing the data.

The Dell server mentioned with a 2620v4, plus RAID controller running the 'Professional +' code does allow for this cam load to be handled.   The choice of running the 'archiving' function comes into play with the 'plus versions' as it will reduce the supported throughput due to sharing of the storage path.

If you are using the older 'Professional' non-plus code base then the storage is best as a single array and volume.

If you are NOT using RAID protection, then pay attention to how many simultaneous usages are supported by the drives.   Do not use consumer drives as they are not made for 24/7 operation.

In the end...when an issue happens...do you want to figure it out yourself or work with someone who knows how to diagnose it already?

 

 

(8)
(9)
Avatar
Mike Dotson
Feb 05, 2018
Formerly of Seneca • IPVMU Certified

With this said, there is a spin on the integrated GFX part that is hinted at in this discussion thread ( Other Milestone discussion ).   This is Client only per the discussion.

At the same time you can see that Intel is now integrating AMD based GFX engines in some of their CPUs ( Example Intel gfx ) so one will have to be careful in CPU selection very soon.

How can you compensate?  One possibility is to use more CPU power at a much greater expense like in the old days.

(4)
Avatar
Christopher Freeman
Feb 19, 2018

Thank You mike 

I did not reallize this until this forum

really appreciate the info. as we have been thinking it was the manufacturer , not the  processor for some time. 

When in fact it is the design intent 

Really makes a difference. 

 

(2)
LC
Lew Cody
Feb 07, 2018
IPVMU Certified

Guys, thanks very much for the feedback - much appreciated!

Points taken. We don't believe Milestone is exploiting anything and they have legitimate reasons to be prudent and to recommend a particular hardware platform.

We are looking at this from two angles

1. We have existing high end i7 'white boxes' with SSD that run 25+ cams (2MP) and have performed flawlessly for 24 months now.

2. There is a $3000 differential in price between a DELL and a 'white box' and we consider that to be a large price difference for (in our view) a less than tangible performance increase.

@Mike Dotson...just curious about your comment re wearing out a CPU - if we run it at <20% utilization, how does one wear out a CPU? Surely it's just a switching circuit at the electron level?

AT this stage it doesn't seem that anyone is suggesting that the red flags go up and the bells start ringing when we suggest the use of a 'white box'. It seems that the approach is 'rather safe than sorry' or anecdotally 'no one has ever been fired for buying IBM'

 

 

(1)
(1)
Avatar
Mike Dotson
Feb 07, 2018
Formerly of Seneca • IPVMU Certified

Indeed if you are running low CPU utilization you will last much longer.

Part of your historical description mentioned pegging the CPU when higher frame rates were used.   My statement is cautionary along that line where your utilization was high and if these i-series CPUs run at 60+% for 24/7--365... there could be an early fail introduced.

Use perfmon to watch how much data is really getting written to the SSD for a 24 hour period and compare that to the endurance value.

Hey...my past life was as an IBM Engineer for many years....we DID do a lot of corner testing on the systems we designed ;-)

(6)
Avatar
Walter Holm
Jun 13, 2018
IPVMU Certified

The Dell servers offer OOB Management features, support contracts (they show up to replace hardware), branding options, etc.

As for trying to stay in spec and build something else, that is reasonable too.  Keep with the Xeon processor and likely ECC ram (less system crashes, for example).  Keep the similar core could and equal to or greater than cache sizes.

I would use SSD as an offload option (with the ability to bypass SSD) as SSDs do fail.  Hot swap drives are also quite useful.

Of course you can do all this with a Supermicro (no onsite support but still, everything else could be cloned).  My experience though, Dell does a great job building their servers so less problems tend to occur within the chassis (physical problems, not necessarily hardware failures).

UM
Undisclosed Manufacturer #2
Jun 13, 2018

[Milestone Employee]

 

A potentially interesting wrinkle is that if you're recording on motion, the system can choose to buffer new video into RAM first (rather than write to a hard drive), analyze it for motion and only write to disk if motion is detected, potentially saving a lot of wear on the SSD. Is it worth it? Certainly debatable, but it can lower the use of the SSD.

Avatar
Mike Dotson
Jun 13, 2018
Formerly of Seneca • IPVMU Certified

To add to the use of RAM, be sure to evaluate if you have enough RAM for this operation as it can add quite a bit for the higher MP streams such that where one could use 16GB before, the prebuffer will demand 24-32GB depending on the streams.   A good high end server should have 32GB now.

(2)
(2)
Avatar
Walter Holm
Jun 14, 2018
IPVMU Certified

Another good reason to use a Xeon, to have more RAM available.  This is also a good reason for Registered and ECC RAM options.

Could also run SSDs in a RAID array with a hot standby but as we know they have a wear limit, it will happen and it's a matter of when.  Having them hotswap is great for supporting the server without taking anything offline...another good thing to mention, most consumer grade RAID doesn't allow hotswap.

Avatar
Jared Tarter
Jun 14, 2018
Milestone Systems

[Milestone Employee]

The amount of RAM used for pre-buffering in RAM is typically fairly minor.  It can be easily calculated based on the bitrate of the streams.  By default, our software pre-buffers 3 seconds.  For example, if you have a 2 Mbps stream (fairly normal for a 1080p @ 10fps), that is the same as 0.25 MB/sec.  So, with a 3 second pre-buffer, each camera would use 0.75 MB of RAM.  With 100 cameras running at 2 Mbps and a 3 second pre-buffer, the increased RAM utilization would be roughly 750 MB.

Even at a higher megapixel camera that would be running something like 8 Mbps, your increased RAM utilization would be roughly 3 GB.  Generally speaking, customers aren't putting 100 cameras running 8 Mbps on a single server.

(4)
Avatar
Mike Dotson
Jun 14, 2018
Formerly of Seneca • IPVMU Certified

I agree on this.   I was blending two memory effects.

The 'needing extra RAM' percentage wise has more to do with the Hardware Acceleration on the server.   On a system with only the Intel Qsync, a 64 channel setup of 4K 14Mbit streams will change from 7GB to 12GB due to HA being active.

A beneficial effect of the new 2018-R2 code that uses the Nvidia cards is that the card memory is used, thus the stress on the system memory is reduced.

U
Undisclosed #3
Jun 13, 2018

Don't most of the servers have redundant and more efficient power supplies? Do consumer grade motherboards offer the ability for redundant hard drives? I'd imagine there is a difference in performance/reliability in the RAID controllers?

Avatar
Walter Holm
Jun 13, 2018
IPVMU Certified

There is very much a difference in RAID quality and performance.  Most consumer grade boards will not do even RAID 5 (only variations of 0+1/1 typically).  Even if they did, performance is no where near the commercial grade hardware.  You can certainly add that level of RAID controller to the consumer hardware though.  However, there is a difference between Xeon and a i5/i7 as well.

(3)
Avatar
Mike Dotson
Jun 14, 2018
Formerly of Seneca • IPVMU Certified

The mobo based RAID5 is a very strange beast with a maximum performance around 100Mbits. 

Two primary factors are lack of dedicated 'engine', thus using the CPU, and the lack of a cache that can be put in writeback mode.   The latter is the most important for performance.

It has also shown to be very slow on the rebuild times, so we no longer advocate R5 on the mobo.

(1)
(2)
Avatar
Luis Carmona
Jun 14, 2018
Geutebruck USA • IPVMU Certified

Having used to owning a computer store and supporting small business networks, I've seen firsthand the performance difference of your onboard and budget RAID controllers. There's a reason there are $100 controllers you can get online and $1000 controllers, just like there are $100 cameras and $800 cameras.

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions