Member Discussion

Arecont Multi Imagers Dropping Communication To Servers

One of our integration partners is trying to get Arecont multi imager camera to connect to a system where both Hanawa and Axis cameras are working

The camera is running the latest version of firmware

The wire shark traces show the camera sending some packets, the server receiving and sending a reply, then the Camera sending replies that never arrive. 

As mentioned, The network is stable as none of the Hanawa or Axis cams are experiencing problems

Anyone have this problem and if so, how was it resolved?

 

 


Anyone have this problem and if so, how was it resolved?

What model? I have a lot of problems with the Arecont AV8185DN, more so than any other model of camera. And yes, even though our streaming code works with other brands, doesn't mean it will work with the Arecont. From memory, these were the problems I had:

1) Appears to emit h264 that crash Intel h.264 IPP decoders on some processors. I do not know if this is Intel at fault or Arecont, but the problem seems to have been there since at least 2009:

UMC H264 decoder crashes on decoding Arecont H264

2) The AV8185DN multi-imager ( at least the one I have on my bench) seems to drop the connections very easily if you connect with a seperate tcp connection for each of the 4 streams (i.e. 4 tcp connections per camera). However they support streaming all 4 video channels over a single tcp (rtsp) connection, and this seems to result in a more stable connection.

3) I had another problem where the tcp connection wouldn't connect, can't remember the details, but I had to change some of our tcp code to so it accepted a packet in a single read, rather than segmented into seperate reads (or it was something like that...). I didn't think we were doing anything wrong there, and got the feeling there were some strange goings-on with their socket code.

You can get it to work, but it requires a lot more work than with other cameras...

...I had to change some of our tcp code to so it accepted a packet in a single read, rather than segmented into seperate reads...

You rewrote the TCP/IP stack because of Arecont? Now that's being accommodating!

No, just a couple of lines of code I had to change, but shouldn't of had to since it worked with the 40+ other cameras I have on my test bench. 

I have spent 4 times as much time on the Arecont driver as the Axis driver, and when these cameras keep going offline, who gets the support call, Arecont or me? I do!

The AV8185DN I have here is really bad and I am getting fed up. As you say the VMSs have to be very accommodating.

To be fair to Arecont, this is an old beaten up camera and perhaps an old model. I am sure their latest cameras are better?

 

I am sure their latest cameras are better?

Not in my experience.

Not in my experience.

If that is the case, then this is the advice I give to Arecont:

I have written 15 or so VMS drivers for different camera manufacturers and so I see cameras from a different perspective to installers, manufactures, managers etc. Some cameras work well with VMSs, most are less than perfect, but the Arecont was the worst...

One thing I notice is that the cameras I like and do not like and pretty much the same as the cameras the installers like. I really like Axis, I do not like Arecont. Therefore, I am thinking that if Arecont changes things I don't like about their cameras, their problems might be solved. I am wondering how many  problems with the cameras are that they just do not work well with third party VMSs?

So from a VMS developers perspective, this is what I think Arecont should do:

Hire a good Windows C++ developer and write an application designed to test each camera model to the limits that I think it needs to handle in order to survive out in the field without doing annoying things that cause the NVRs to keep loosing the connection etc.

The application should do the following:

1) Connect to the camera using the following protocols using open source rtsp clients available on the web e.g. live555, and as many others that you can find as these are generally the ones NVR use, although not always. The application should then be able to connect to the cameras using the following protocols.

  1.  tcp over rtsp (mandatory)
  2.  udp over rtsp (optional)
  3.  rtsp tunnerled through http (mandatory)
  4.  MJPEG over HTTP (mandatory)

IMO, Areconts proprietary protocol shouldn't be needed these days.

2) For the H264 decoders use:

  1. Intel IPP H.264 decoders
  2. Intel media SDK hardware accelerated decoders
  3. Optionally NVIDIAs GPU base decoders

In my experience the h264 emitted from at least some Arecont cameras cause problems with all three. VMS companies generally do not write their own decoders, they are most likely to use the Intel decoders.

3) Test all decoders on the full range of Intel CPUs, from Celerons to Cannonlake. The Intel decoders load different code depending on the CPU type. I have seen h264 emitted from Arecont cameras crash the Intel decoders on Pentiums (really really annoying). On generation 3 i7s it causes the decoders to throw memory exceptions that are at least caught. I do not think they crash in this case, but you still loose frames.

4) Test using different combinations of h264 configuration parameters. e.g. the crash I mentioned in (3) was somewhat eliminated if I force the cameras to use constant bitrate h.264 instead of variable when the VMS connects.

5) At an absolute minimum, simultaneously create a tcp connection for each imager and each stream. So if it is a x4 imager and supports dual streaming there will be 8 connections. Do this while having at least 2 web pages open, and make sure the cameras can handle this without dropping connection and crashing. This is at a minimum. The Arecont 8185DN that I have can't handle this, it reboots, looses connections and crashes all over the place. Might be stable for 2 minutes, and then drops connections again. By comparison, some Axis cameras can handle 20 concurrent tcp connections for example. 

6) Test this on a network, or simulated network that is noisy, has large packet loss etc, and at the same time send corrupt data to the cameras, drop connections unexpectedly etc. 

7) when the test application sends tcp packets to the camera, break them up with small delays. e.g. if the camera is expecting 100 bytes, send 10 and wait a bit, send 70 and wait a bit then send the remaining 20. When receiving a reply from the camera do the same. Call the socket receive call and retrieve 10 bytes and wait a bit, then get the rest...

8) Insert random timing delays between receiving frames, and parts of frames from the camera. This puts memory pressure on the camera, many don't handle this well. Make sure in such cases that the camera can throttle down, e.g. drop p frames until the next I frame etc, drop the quality etc. I have seen even some axis cameras slow down and freeze under this test.

9) Create and drop 1000s of connection to the cameras over a period of time to test for memory leaks. There are undoubtly cameras out there (not talking about arecont in this case) that will slow down until they eventually freeze or reboot. 

9) Run the application for 1 week and make sure there are no reboots at all, no memory leaks and an acceptable number of unaccountable lost connections. 

If the camera can not handle this test, keep improving the software, and if necessary the hardware (i.e. processing power and RAM) until it does work. 

NOTICE: This comment has been moved to its own discussion: How To Make IP Cameras Work Better With VMS Software, From A Developer

Really interesting information. I'm not a programmer, but I like learning about the nitty-gritty of how these work.

This reminds me of a problem we had getting 20+ frames per second out of an Arecont several years ago. This was a box camera. Turning off the Sony cameras would allow the camera to go full frame rate. The Sony's never had a problem doing full framerate. Bandwidth should not have been an issue and it was not a lot of cameras.

What we eventually found as a work around was putting the Arecont on it's own network connection direct to the NVR. It did not seem to like being on a network with other cameras.

Large enterprise customer here...

This all sounds familiar. I have wrestled with Arecont for almost 5 years.

After years of unresolved service calls by my integrator, failed tech support calls with Arecont and discussions with local reps who denied the issues with these cameras, I decided to replace all Arecont cameras with Axis Q3709.

So far, no problems at all with my Axis 180 degree cameras.

The wire shark traces show the camera sending some packets, the server receiving and sending a reply, then the Camera sending replies that never arrive.

Firewall?