Digital Watchdog Blackjack Cube Multi-Server

Full Disclosure- I work for Digital Watchdog

Hi John

i just saw your post this morning. Please give me a call on my cell 813-545-9549 so we can discuss your application.

Thanks,

Patrick

John, what struggles are you having specifically?

Good morning. The application is a large water treatment facility with numerous buildings on the plant. The Network is a fiber backbone, each one of the buildings have a variety of Cisco switches. Most of the switches are 4/8 port hardend variety.

We have 2 DW Blackjack Cubes (Windows 7) that should, theoretically, be merged. Presently, they are not, as a merged system seems to contribute to unstable operation.

The system consists of 17 DW (5MP) cameras and 10 non-DW cameras. The heart of the issue is packet loss. We are seeing major packet loss only on the DW cameras. 5 of the non-DW cameras are Arecont 180's (20MP) streaming full resolution...with no packet loss.

The topology of the network has most cameras making multiple hops across several switches before it gets to the server. Customers IT department has worked very hard to insure that the network is not the issue. If the non-DW cameras were experiencing packet loss, I would question this.

We had achieved stable operation when we dumbed down the DW cameras to 1080p but the cameras stopped recording, because they will not do event recording when dumbed down. When I set them to continuos recording, packet loss resumed, but only after I dragged more than one or two cameras to the desktop for live view. The more cameras I view, the worse the packet loss becomes.

I have been working with DW for a couple weeks now and have involved a very knowledgable DW rep, but I figured I would pick some more brains.

Presently all frame rates are at 6IPS. 17 DW cams at 5MP; 5 Arecont 180's at 20MP; and 5 1.3MP cameras. All 17 DW cams are on one server. This (DW camera) server is connected to the camera LAN (all cameras on the same LAN) via NIC 1 and to the corporate network, to enable remote viewing, via NIC 2. All other cams are on the 2nd server (non-DW cam server). No packet loss on the 2nd server. As of midnight to 11:30AM MST, the DW camera server has logged 10,000 packet loss events, while the non-DW server has none.

Presently, both servers are unmerged, which obviously makes the non-DW server unviewable remotely, because it is only connected to the camera LAN.

Are all the DW cams having packet losses? Or just a few ? Try swapping the NIC configurations since you have two nics on the server. Cam --> Nic 1 , Wan --> Nic 2 etc.

Thanks to everyone and their responses.

UPDATE:

I have not worked on this issue for the last 4 days, as it had simply taken me away from all my other work and I needed to catch up. Today my DW rep and I worked on this issue all day.

What we found was this:

More than likely all cameras were experiencing packet loss; the DW cameras simply displayed and logged the packet loss issues because of their level of software integration with the DW Cubes and Spectrum software. We discovered this by looking at a known packet loss event in the system log and reviewed the recorded video. On play back we noticed a pause of about 4 seconds right when the packet loss incident happened, which was expected. Out of curiosity, we checked an Arecont camera at the same time and found the same pause...

All this said, while we had concerns about the network, we think the problem may be in Live View of the cameras directly from the server. We noted that when the servers were not running Spectrum, packet loss seemed to disappear. Media server was still running, of course, but the actual Spectrum IPVMS software was not open. (In this installation, the building that houses the two servers is manned 24/7 with two 40" monitors displaying live video feeds that come directly off of the servers; while two remote PC's in other buildings were running client)

We went to a couple PC's running the client software and monitored the cameras and network. NO PACKET LOSS. We put all cameras on the clients in live view and still no packet loss.

Lessons learned: In this particular network environment, the servers can only run the media server but appear to not be able to view live video. We installed a small, but robust, PC in the same cabinet with the servers, opened up the client software and ran the two 40" monitors of off this client PC.

I will be back on site tomorrow to check the system logs and see if we made it throughout the night without any packet loss....

Cross your fingers....