Digital Watchdog Blackjack Cube Multi-Server

I am curious if any members have attempted to implement a DW IP surveillance system using multiple DW Cubes and DW megapixel cameras? I am having some struggles with a system and am looking for some input.


Full Disclosure- I work for Digital Watchdog

Hi John

i just saw your post this morning. Please give me a call on my cell 813-545-9549 so we can discuss your application.

Thanks,

Patrick

Patrick. I will try calling you today if you are available. Thanks so much for your response!

John, what struggles are you having specifically?

Good morning. The application is a large water treatment facility with numerous buildings on the plant. The Network is a fiber backbone, each one of the buildings have a variety of Cisco switches. Most of the switches are 4/8 port hardend variety.

We have 2 DW Blackjack Cubes (Windows 7) that should, theoretically, be merged. Presently, they are not, as a merged system seems to contribute to unstable operation.

The system consists of 17 DW (5MP) cameras and 10 non-DW cameras. The heart of the issue is packet loss. We are seeing major packet loss only on the DW cameras. 5 of the non-DW cameras are Arecont 180's (20MP) streaming full resolution...with no packet loss.

The topology of the network has most cameras making multiple hops across several switches before it gets to the server. Customers IT department has worked very hard to insure that the network is not the issue. If the non-DW cameras were experiencing packet loss, I would question this.

We had achieved stable operation when we dumbed down the DW cameras to 1080p but the cameras stopped recording, because they will not do event recording when dumbed down. When I set them to continuos recording, packet loss resumed, but only after I dragged more than one or two cameras to the desktop for live view. The more cameras I view, the worse the packet loss becomes.

I have been working with DW for a couple weeks now and have involved a very knowledgable DW rep, but I figured I would pick some more brains.

We have 2 DW Blackjack Cubes (Windows 7) that should, theoretically, be merged. Presently, they are not, as a merged system seems to contribute to unstable operation.

17 @ 5MP + 5 @ 20MP + 5 @ ?MP, depending on the frame rate, is not an insignificant load. Can you provide more details on the hardware configurations of the servers, and which way the cameras are split between them?

Presently all frame rates are at 6IPS. 17 DW cams at 5MP; 5 Arecont 180's at 20MP; and 5 1.3MP cameras. All 17 DW cams are on one server. This (DW camera) server is connected to the camera LAN (all cameras on the same LAN) via NIC 1 and to the corporate network, to enable remote viewing, via NIC 2. All other cams are on the 2nd server (non-DW cam server). No packet loss on the 2nd server. As of midnight to 11:30AM MST, the DW camera server has logged 10,000 packet loss events, while the non-DW server has none.

Presently, both servers are unmerged, which obviously makes the non-DW server unviewable remotely, because it is only connected to the camera LAN.

Just to throw out, how much is the DW server being viewed? Since that one is the only one providing outgoing streams, have you ruled out that it might be getting swamped with outbound requests, causing it to fall behind?

What's the config of the servers, identical?

What are the performance stats for CPU and memory utilization?

Are there any clues in the distribution of errors logged? Time of day, interface, endpoint, etc.

Are all the DW cams having packet losses? Or just a few ? Try swapping the NIC configurations since you have two nics on the server. Cam --> Nic 1 , Wan --> Nic 2 etc.

Thanks to everyone and their responses.

UPDATE:

I have not worked on this issue for the last 4 days, as it had simply taken me away from all my other work and I needed to catch up. Today my DW rep and I worked on this issue all day.

What we found was this:

More than likely all cameras were experiencing packet loss; the DW cameras simply displayed and logged the packet loss issues because of their level of software integration with the DW Cubes and Spectrum software. We discovered this by looking at a known packet loss event in the system log and reviewed the recorded video. On play back we noticed a pause of about 4 seconds right when the packet loss incident happened, which was expected. Out of curiosity, we checked an Arecont camera at the same time and found the same pause...

All this said, while we had concerns about the network, we think the problem may be in Live View of the cameras directly from the server. We noted that when the servers were not running Spectrum, packet loss seemed to disappear. Media server was still running, of course, but the actual Spectrum IPVMS software was not open. (In this installation, the building that houses the two servers is manned 24/7 with two 40" monitors displaying live video feeds that come directly off of the servers; while two remote PC's in other buildings were running client)

We went to a couple PC's running the client software and monitored the cameras and network. NO PACKET LOSS. We put all cameras on the clients in live view and still no packet loss.

Lessons learned: In this particular network environment, the servers can only run the media server but appear to not be able to view live video. We installed a small, but robust, PC in the same cabinet with the servers, opened up the client software and ran the two 40" monitors of off this client PC.

I will be back on site tomorrow to check the system logs and see if we made it throughout the night without any packet loss....

Cross your fingers....

John

thanks for the update.

Regardless of resolution.. and current situation, Vincent in the thread above did give you good advice: DW cube has 2 NICs. One is embedded to mother board another is not.
The one which is NOT should be pointed to the cameras.

some additional technical explanation...

Mother board NIC shares CPU to get packets, unlike discrete one.
So if CPU is busy( by client for example) and you use embedded NIC to get video out of cameras it can have issues getting data.

On other hand most VMSes get video on RTP over TCP. So if single packet is lost it should not be an issue for a camera to resend the packet. However if there are multiple packets lost, camera will drop some frames.


Again, you simple solution could be just "swapping cabel"
Actually I still recommend doing this, regardless of the fact that it works as is. It's like MUST do thing. Unless it's already this way:-)