Storage For Milestone Solution [Urgent]

Guys This is a followup for the thread I posted last time.

And now it seems clearer. the solution will go as follows

a. Samsung Camera for 20 REMOTE branches and each branch has 15 cameras in 1920x1080. Maximum bandwidth taken is 2.5Mbps according to our vendor

b. Samsung Camera for HQ has about 200 cameras

c. Milestone Xprotect Corporate

d. Milestone recording server and mgmt server at both remote site and HQ

e. External storage which could be EMC isilon or Netapp E2700A

f. The total archiving size for storage is 649TB

I want to know which one provides with better interoperablity and performance with Milestone according to the scenario as mentioned above? Is Netapp E2700A working well with Milestone in a deployment scale of total 500 cameras of which 300 of them are remotely connected and 200 of them are locally connected.

In addition, here's the info for the storage uplink but I have concern that if 1Gbps is enough for archiving as follows

a. the uplink port from isilon switch to server switch where the milestone recording server will be connected to is 1Gbps

b. the uplink port from Netapp E2700A is 8Gbps FC in that the milestone recording server must be equipped with a FC interface card.

The total bandwidth subscribed for all 20 branches will be 600Mbps which support for 300 cameras. And the 200 cameras stream at HQ which is not optimized takes up to 6Mbps each so in total of 600Mbps + 1.2Gbps = 1.8Gbps. So from a network perspective the central Milestone recording server must be able to support at least 2Gbps(1Gbp x 2 in portchannel) or using FC. So 1Gpbs uplink sized by EMC is definitely not enough. Anyone has different idea than mine?

So in short I'd like to have your advice on which one works better with Milestone

EMC isilon or Netapp E2700

Appreciate your attentions!

Thanks,
Michael


Michael,

From what you have shared with us, this is a poorly thought out 'solution'. I urge you to do more research and testing before finalizing the components.

For instance, in the previous thread, you mentioned, "it turns out after conducting the failover to edge storage which is the SD card on Samsung Camera it seems taking hours just to sync back about 5mins of video recording for 20 cameras at one branch. It's quite unacceptable and probably we are looking into other brand to replace it." When I asked about this, you acknowledged, "yes you got it right the problem could be either from the SD card from Sandisk, parts from Samsung Camera or milestone recording server but so far we haven't come to conclusion yet." So since you are specifying Samsung and Milestone here, have you solved the problem?

Another example, "The total archiving size for storage is 649TB" Are you sure? In the previous discussion, you noted, "1 year retention for HQ stroage is required." With 800 cameras, you might need far more. Are you pruning storage? How are you getting to 649TB instead of 2PB or 7PB, etc.?

Also, when you are specifying throughput into a server or storage array, you need to consider max throughput. Have you tested the bandwidth consumption of the models you are using? Bandwidth needs go up generally during busy periods of the day or at night (e.g., Testing Bandwidth vs Low Light). Have you factored that in? Your average throughput needs might be X, but your max throughput might be 3x.

As for "EMC isilon or Netapp E2700", spend some time and money researching this. You are going to be spending at least hundreds of thousands of dollars here. Ask Milestone, EMC and Netapp for end user references who are using those combinations. Talk to the end user's IT department or integrator to get more details about any specific low-level problems that might be ocurring, etc.

Dear John,

Thanks a lot for your genuine advice.

The project was suddenly pushed out due to some compliance reason from the external auditing they just conducted and it's going to happen in a month time. Everything has to be quickly tested and verified hence we conducted Samsung + Milstone and collected bandwidth info which was about optimized 2.5Mbps for remote branch office use. I can say it's not a figure coming out from a calculaor but was verified by POC done last week. However unfortunately the vendor is still undergonig inspection of the slowness issue for edge storage failover:( and there's no resolution for it yet.

And due to the pressure from the board of mgmt and external auditor the client somehow becomes desperate for a solution that can just simply work fine. I understand from a vendor perspective we should take a step back and think thoroughly but we are somehow really running out of time as well and have to rush it out in about 2 weeks time.

The practice of this combination is really rare and it's really difficult to judge which one is better at this moment So that's why I would seek the help from here and hope I can get a firm answer to that.

Thanks again for sincere help!

"I can say it's not a figure coming out from a calculaor but was verified by POC done last week"

2.5Mb/s is average or max bitrate? You tried it at different locations, different times of day? Do you expect to record at night? It's a bank so motion only then or? All of these things have a big impact on both throughput and storage.

"And due to the pressure from the board of mgmt and external auditor the client somehow becomes desperate for a solution that can just simply work fine."

Are they going to hold you responsible for making this rush job work fine? If not, then sure it's on them. But if it is on you, I urge you to pad your quote significantly (extra 30% or 50%) to factor in the inevitable unforeseen problems that are going to come from rushing through such a large and complex project.

well I think 650TB is a little low for 1 year retention for that much video, seems like it closer to 1PB if there is some file storage compression being done. if I had that many remote sites I would look into a cluster style storage solution, that way you can spread all of the risk out every where. but if one centralized storage is what you want network bandwidth is not your main issue its going to be I/O or writing to disk. EMC and Netapp will sell you anything tha can accomplish this but it sounds like having central storage is not the problem.

also how much bandwidth do you have available on these remote sites?. that could be a reason why it takes so long for a SD card backup, or it could be the settings on the FTP function of the cameras. some cameras have a file transfer limit setting of 20 KB or 20 MB or it could be a setting in milestone, how all this is configured can be an issue that wont be resolved even if throw money at a storage solution.

you could FTP into the camera with filezilla(or any ftp program) and see what the transfer rate from the SD card is, that would give more infomation to narrow down some of your issues.

so far as pairing up external/ 3rd party storage, it quite common that why VMS's like milestone are flexible in their storage options(CIFS, Block, etc) its just rare(short of the government) that it would be on this size.

A bank wanting to record video from all these branches across Internet connections and storing one year video.... you know, it sounds strangly like a job we were looking at and we showed them calculations that pretty much demonstrated why it was a bad idea. But they didn't like our cost for storage at each branch and decided it could be done cheaper and the way they wanted it. We didn't walk, but stuck to what we knew would work, not try and shoe horn something into what they thought would work or wishful thinking. So they turned us down.

I know you can't disclose their name, but it'd be real interesting to know who the customer is.

So in short I'd like to have your advice on which one works better with Milestone EMC isilon or Netapp E2700...

Michael, I think it's wise to look at the specific application and proposed solutions like you have, but at some point and in the absence of any red flags, you need to compare Nedap and EMC on generic storage terms.

Unless you have some particular reason to believe that one outperforms the other when used in video recorders? Is there some technical point between the two that has you concerned?

If not, then you should shift the due diligence to comparing the storage offerings on their own terms. IMHO.

Disagree.

There very well may be differences in how the 2 connect and deal with the load from Milestone. So while he should certainly review the generic elements, that's why I am recommending that he get specific feedback from existing Milestone users (of hopefully similar size / scale of what he is proposing to do).

Disagree. There very well may be differences in how the 2 connect and deal with the load from Milestone.

Agree. There may well be. And the feedback from specific use cases could be key. But on IPVM at least, I don't hear anyone throwing up any cautionary advice either way. Which makes me think that it might be hard to come up with a significant number of survellience data points, specific to the exact models desired.

Especially now that there is a time constraint. So it's a practical issue.

If there was a THEORY proposed as to what might impact, e.g. caching strategies, that would be helpful.

But to take all the remaining time trying to find the perfect surveillance reference installation and then make ones decision based on whatever specific but sparse data points found, might be unwise. Again, in the absence of some suspected issue.

I won't say "Storage is just Storage" because I agree it's not always the case. But if the both manufacturers are saying they can perform adequately, and with little time remaining, you do not want to overlook the other things about the offerings that are not industry specific.

<edit> not

"But on IPVM at least, I don't hear anyone throwing up any cautionary advice either way."

That's because the number of people in the world connecting Milestone to EMC or Netapp are fairly low. It's certainly been done, I am just saying a discussion is unlikely to get many people who have integrated both EMC and Netapp to Milestone in a large scale production environment.

"But to take all the remaining time trying to find the perfect surveillance reference installation"

What is Milestone doing here? They surely have time to find a reference for a job this large that is relevant. Milestone benefits from a good recommendation here as well as if there are performance issues, Milestone support will have to deal with this.

Michael, have you gotten any recommendation on the storage choice from Milestone directly?

I agree with the others here; not sure how the 649GB was calculated (please share), but...

Assuming basic requirements of 12 hours of continuous recording, 10fps, average frame size of 34KB (This might be a little high, but just an assumption) assuming H.264 VBR, 1080P resolution, at 260 days for 300 cameras, that puts the storage at more like 1.1 Petabyte and required bandwidth at ~800Mbps aggregate at HQ for just the remote sites. You still need to consider the storage for HQ cameras, so a total of ~1.9 Petabyte of storage will be required assuming the listed assumed requirements; bandwidth on the local network to the SAN will be on the upwards of about ~1.4Gbps, assuming average bandwidth per camera is 2.7Mbps (This will vary on scene).

You will NOT be able to use Etherchannel; Etherchannel does not move over any traffic load when link saturation occurs. Traffic through a specific link is based upon a hash on the mode being used. You will more than likely have to utilize a direct FC interface card on the Milestone and connect to the FC port on the SAN.

At this point, I do not believe that an E2700 Netapp device will be adequate for the solution based on the assumptions (much of that was not provided). You are definitely going to need expansion shelves to whatever device/vendor you go with due to the amount of storage required.

Something else to consider:


Internet bandwidth costs: ~$50,000/Yr

HQ - ~1000Mbps: $3K/mo (+/- a few hundred dollars); depending on vendor and location, there might be an affordable solution, but here in Hawaii, business bandwidth is expensive.


Business Fiber Internet 50/50 ~$100.00 (20 sites, 2K/Mo)

~50K a year just in bandwidth charges

Storage costs: ~$250,000 (Includes support/warranty costs for 3 years)

Initial Cost of a 1 Petabyte storage solution can cost you nearly $250K - $2.8 Million dependent on the vendor and hardware you choose for the storage solution. The raw drives alone can cost you roughly $43,000 per Petabyte; if you shop around, you can maybe get to ~$38,000. I definitely would not stripe that to a Raid0, so with Raid5, you are looking at ~$114,000 - ~$139,000. If considering Raid6 or 10, that will increase the cost of the drives and hardware.

You will definitely need to contact Pre-sales support at Milestone to ensure that you have adequate server resources to support the solution as already mentioned.

That why I said a cluster style storage like openstack or ceph/gluster would be better you could scatter your storage across all of your sites so if a drive dies anywhere it would be a simple matter of turning on a hot spare or replaceing the drive.

but you would have to pay some one to handle this. Redhat and Ubuntu have services but this is on top hardware. still looking at some serious cash though.

so far as the storage, milestone corporate will store to any of the industry standards netapp and EMC included, how to configure this will be is something you need to consult milestone support on.

a little cheaper but still reliable would be imations storage line "nexsan" still looking in the $50,000 -$100,000 though

I can comment on the Netapp E2700 product as I'm the Netapp Sr. Solution Architect for Video Surveillance. I agree with much of the discussion so far in terms of knowing as much as possible about the data rates from the cameras and your retention requirements.

Netapp has done extensive testing with Milestone and published several certification reports and whitepapers discussings best practices and reference architectures.

The E2700 family has more than enough throughput to handle most any surveillance workload which is overwhelmingly (90%+) random writes. We routienly deploy systems with over 1.2Gbps throughput across as few as (30) disk drives. The limiting factor is the number of drives, not the controllers or host interfaces (SAS3, 1/10Gb iSCSI or 8Gb Fibre Channel). We can scale linearly in (30) drive pools each driving 1.2+Gbps up to the (180) drive limit for the 2700 model, which can also be field upgraded in as little as 30 minutes to a 5600 Series controller that doubles the throughput and the number of supported drives to (360).

The issue in enterprise class surveillance at scale is archive density - how much provisioned storage can we get into the densest packaging - without scarficing reliability, fault tollerenace, ease of use and price/perfromance. I'd be happy to take this discussion off-line with you or anyone and make sure you get the local resources & E-Series references in surveillance to help you make an informed decision.

Ted Hayduk, Netapp

So in short I'd like to have your advice on which one works better with Milestone, EMC Isilon or Netapp E2700.

From EMC/Milestone Architecture Document for Xprotect Corporate

Hi Michael, from the EMC perspective, I can provide some insight as I'm the Solutions Architect for the global surveillance practice. EMC and Milestone have a very tight relationship in which we've successfully deployed several solutions for a variety of different customers on EMC Isilon. I'm going to try to be as consise as possible, but this response is going to be a little long (apologies in advance).

Some background - Isilon is a clustered file system in which each server (we call this a node) interacts with the other nodes to aggregate their combined CPU, memory, networking and storage into a single storage volume that is presented to the client as a network share - in the case of Milestone this is an SMB share. The benefit here is what we call 'scale-out'. That is, for each node you add, you're not just adding storage, but you're adding CPU, memory, and networking as well. One of the benefits here is that you can 'buy as you grow' - so purchase what you need now and seamlessly add additional storage in the form of nodes when they are needed later. This is an important point. When adding storage, there is no drive re-mapping, no LUN's to sort out, no additional volumes - the drive letter or UNC path that Milestone writes to can literally grow in size.

To address the point above regarding avoiding using Isilon for the LiveDB, we do recommend that the live DB be stored on some kind of DAS (which can be SAN, but can also be the disk which is shipped with the server). This is due to the highly transactional nature of the DB. Video, of course, is going to demand the most capacity and this is the part that is stored to Isilon. We call this "tier 2". Video lands on the server in some form of DAS (or SAN) then after some amount of time (usually a matter of hours, but up to 1 day) the milestore server then 'archives' the video off to the network share. We've successfully deployed such architectures with Milestone in a number of publically referencable cases.

EMC publishes our partner validations, as performed by out dedicated surveillance labs (one in Bangalore and one in North Carolina) on the following site http://www.emc.com/storage/video-surveillance-storage-partner-validations.htm where you can find details about our test regimes for a number of surveillance partners. Another resource we can offer is the ability to simulate planned deployments so that we can show you what the designed environment will look like with simulated cameras and actual surviellance software. I presented a similar demonstration at ASIS, in which we had 1.5PB of storage in 20 rack units, streaming 1000 cameras at 1Gbps. Part of the demo was to add and remove a node simulating node failure and adding new capacity resulting in no impact to recorded video (i.e. everything continues to work).

If you'd like to discuss in more detail, please feel free to reach out.

Thanks

Brent Cowing (brent.cowing@emc.com)