Subscriber Discussion

Axis And Exacq Connectivity Problems - Both Say Its The Others Fault

UI
Undisclosed Integrator #1
Nov 29, 2016

Anyone else having connection issues post Exacq 7.4.2 with Axis cameras?  We are seeing consistent drops and camera web server freezes.  Exacq says its Axis fault and Axis says Exacq. Neither of them will own the problem.  Both say its a driver issue and compatibility between the software. 

 

Is this a new issue the Milestone Merger has brought on? 

UM
Undisclosed Manufacturer #2
Nov 29, 2016

I'm not sure if it could be related to the same issue you are having with Exacq, but we had an issue not too long ago with connection dropping from Axis cameras. After extensive investigation we found out that after Axis implemented a new RTSP server along with a new HTTP Server (I don't remember which firmware version), the RTSP server would use the control character "$" in the SessionID field.

The character "$" is used as a control character inside the stream when streaming using RTSP over TCP to split between internal communication channels.

When Axis camera was sending this character inside the SessionID, it could affect the stream parsing, and a connection loss could happen.

We have identified the same behavior when viewing the streams using VLC as its parser would also be "confused".

We have reported to Axis and they have made a temporary fix for the cameras of our client, but they said they would merge the fixes into 6.40 firmware for all cameras.

Again, I'm not sure it would be the same problem, but it would be worth trying to see if the latest firmware version would fix it.

(1)
U
Undisclosed #4
Nov 29, 2016
IPVMU Certified

...RTSP over TCP...

Do you mean RTSP over HTTP?

UM
Undisclosed Manufacturer #2
Nov 29, 2016

Both

RTSP over HTTP over TCP
RTSP over TCP

Because you can also transmit RTSP data over UDP, in that case there is no "$" delimiter because each channel is transmitted over a different UDP port, hence no problems...

U
Undisclosed #4
Nov 29, 2016
IPVMU Certified

But the $ delimiter is used when you need to interleave media and control channels on the same stream.

An ordinary RTSP/TCP connection should have independent RTP ports, and therefore doesn't need to interleave or even want to because of the performance/control considerations, no?

UM
Undisclosed Manufacturer #2
Nov 29, 2016

You may be talking about RTSP over UDP, in that case you have different RTP ports (UDP) and each stream (video, audio, control) are sent to different UDP ports

When transmitting everything over TCP (usually everything on port 554), communication is interleaved, video, audio, control (RTCP), everything goes into the same stream, and there is a control header that contains the "$" (0x24), channel identification and length, and all streams are interleaved into the same transmission...

U
Undisclosed #4
Nov 29, 2016
IPVMU Certified

When transmitting everything over TCP (usually everything on port 554), communication is interleaved, video, audio, control (RTCP), everything goes into the same stream...

I'm aware you can interleave with TCP, I'm not sure that you have to.

Regradless, Axis must be on the lookout for $'s, so I see your point.

UM
Undisclosed Manufacturer #3
Nov 29, 2016

RTSP over TCP is a thing -- interleaving RTP and RTCP packets over the RTSP TCP control channel, i.e. 554/tcp

RTSP over HTTP is really RTSP over TCP ... over HTTP. Another layer of nonsense.

(1)
(1)
UI
Undisclosed Integrator #1
Nov 29, 2016

Thanks for the info, finding a $ inside all that must of been fun. What camera models were you running that were approaching 6.4 firmware? Most of their models haven't broken into the 6.x range all. With all the .x.x.x trailers I could be waiting along time!

UM
Undisclosed Manufacturer #2
Nov 29, 2016

indeed, it was really "fun" :)
They were using Q1635 and P1405 but I'm pretty sure it affects all cameras after the firmware where they changed the RTSP and HTTP server some time ago.

They sent me a fixed firmware 6.30 for Q1635 and a fixed 5.80 for P1405 but they said it would be merged in 6.40

It is a very specific problem and actually would only happen after a camera restart, it looks like the camera would chose to add the "$" character to the SessionID if a connection was made shortly after initializing it. So that is why I think it might not be related to the issue you had. But this problem was visible in VLC as well, creating all sorts of artifacts in the image in VLC

UI
Undisclosed Integrator #1
Nov 29, 2016

You may be right in not the same thing, but exacq is going to look at your suggestion. I should correct my statement in that I do not have "Connectivity" issues, I have "Disconnect" issues. In that the cameras connect just fine but randomly disconnect and lock up the device until disconnected from exacq and a period of time lapses until the device can be reset.

UM
Undisclosed Manufacturer #3
Nov 29, 2016

I can't speak to Axis integration with Exacq, but I've seen increasing amounts of issues with Axis video streams in general. There was a time when you could throw any combination of settings at an Axis camera, and its RTSP stream would always just work. I don't think that's the case any more.

Pulling video from Axis cameras via ONVIF, we'll often see ONVIF video streams just stop sending usable data. The VMS can't record the stream, third-party video tools (VLC, etc.) can't play the stream -- the ONVIF interface is basically bricked until you do a factory reset on the camera. Sometimes this affects streams pulled using the Axis native interface (VAPIX) as well. I've seen similar issues with ONVIF credentials on Axis cameras; occasionally valid ONVIF credentials just stop working until you do a factory reset.

I've also increasingly been hearing a lot of obnoxious anti-ONVIF propaganda from Axis salespeople. "For some reason the ONVIF stream never seems to look as good as our native stream." Pure conspiracy theory bull****. Whether you're using ONVIF or VAPIX, the capture, encoding, streaming, etc. pipelines on the camera are all the same -- it's just a different configuration API and RTSP URL.

(3)
JH
John Honovich
Nov 29, 2016
IPVM

#1, #3, thanks for the feedback, I have forwarded this to Axis (and Exacq). Will provide feedback if / as I receive.

(2)
UI
Undisclosed Integrator #1
Nov 30, 2016

There are many reasons, to choose top tier products from industry leaders, Today it is Both exacq and Axis stepping up their support efforts and are working together to troubleshoot with us.

Hope to have a resolution soon, and will update if we come to a conclusion.

(1)
Avatar
Ryan Hulse
Dec 13, 2016

Undisclosed Integrator may provide more details, but this issue appears resolved with a firmware upgrade to the encoders that also upgraded the PTZ driver.

(1)
JH
John Honovich
Dec 13, 2016
IPVM

Ryan, thanks. Can you share any tech details about what fixed it?

Avatar
Ryan Hulse
Dec 13, 2016

I am one step too many removed to say exactly. I do know that the end-customer, integrator, Axis Support, Exacq Support and one of our Product Managers conducted a series of troubleshooting calls over the last 13 days to continually narrow down the root cause. Ultimately a change of the firmware in the encoders resulted in the issue stopping. We are continuing the conversation with Axis to understand exactly what the root cause is.

UI
Undisclosed Integrator #1
Dec 13, 2016

The tangled details of open access! To say the least we are very great full we choose to do business with Exacq and Axis as their support when needed really comes through. A major detail that is hard to come by when dealing with some 3rd parties compatibility issues. Thanks Ryan for stepping in on this one, it really turns out it was more Axis than exacq.

This was a different problem than what has been occurring in other installs with intermittent outages. Customer has some really old PTZ cameras and had a pre-release of a later supported PTZ driver file on Axis Q7401 encoders. The encoders only show the version number "4.04" with the PTZ driver however there is a signature appended to the actual file name that signifies a different build #.

Key learning here for us is regardless if the encoders shows a version that matches what the release version states as the latest version, try the download on the website anyway as it could still be different than what was uploaded to the device.

I must mention that these are really old cameras, and back then I am not sure what was protocol for releasing drivers, but it seams as though the customer obtained the driver file from the camera manufacturer directly as Axis shows no support for them at the time of install. At some point Axis obtained the ptz driver file and listed it on their support site with identical version numbers. The camera manufacture has long gone under and should not pose a problem for anyone anymore.

Thanks again to both Axis and Exacq for helping us troubleshoot this site and we will revisit our intermittent outages at a later date.

(3)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions