I dont work with the tech team directly but they have been having this issue and would appreciate any feedback that i could possibly share with them . A 2000 plus camera network -dedicated -that had a smooth transition to a core switch upgrade about two months ago. Then it started happening..delays between swithching cameras..5-8 seconds long..black screen in between. What is weird is that it is only happening with certain cameras that may or may not be on the same switch/vlan. Has anyone seen this issue before . They are still troubleshooting and can't seem to resolve.
Subscriber Discussion
5-8 Second Delay When Switching Between Cameras?
U
Undisclosed #1
Aug 24, 2016Hey Ravi,
Are the streams that are slow when switching the same ones being recorded, or a sub stream?
Is the viewing network seperate from the camera network?
How many and what is the usage pattern of the viewing clients?
UM
Undisclosed Manufacturer #2
Aug 24, 2016The delay could be that the client UI has to wait for the I-frame from the camera before it can start decoding all the subsequent frames. So if the camera is only sending an I-frame every 5 - 8 seconds (for example), then you'll have a delay. Two things to look at:
1) GOP/GOV setting in the camera configuration. If it is high, could try setting it to a lower value.
2) If smart codecs are enabled e.g. h264+, Zipstream... this can also cause I-frames to be delayed, especially if there is no motion on the camera at the time when you are switching. In this case you could try disabling the smart codec.
Having said that, if that were the cause it probably would always have done this, but you are saying it only just start to happen?
U
Undisclosed #1
Aug 24, 2016The delay could be that the client UI has to wait for the I-frame from the camera before it can start decoding all the subsequent frames.
That's interesting. Do you think it changes anything if the stream is coming from a VMS that is also recording the stream, since the VMS could have the I-frame handy?
UM
Undisclosed Manufacturer #2
Aug 24, 2016It depends how the VMS is implemented but if it is the recording stream, then most likely it is just directly forwarding the frames it gets from the camera. To decode the current frame, the Client UI would have to decode the previous I-frame and all the subsequent p/b frames, and if the gap between I-frames is 5 to 8 seconds, this could be between 50 to 200 frames depending on the frame rate. I doubt the VMS would send these previous frames to reduce the delay, and will just expect the client to wait. Keep in mind that traditionally, I-frames are every second or two so the wait is not normally that long. Again, smart codecs can cause the delay to be much longer in some cases - it is the price you pay for increased storage... Using MJPEG for the viewing stream may also fix the problem as they do not use I-frames.
Bd
Bas de Leeuw
Aug 24, 2016Hi Ravi,
A few suggestions or things to check:
- Are the problematic camera's using higher resolutions/bandwith
- The new switch might somewhere limit the bandwidth (specific port limit that was not the case before)
Regards,
Bas de Leeuw
Ensura.com
Marc Pichaud
Aug 24, 2016I agree with Bas. It was working, it so no more after the upgrade, so probably they have setup some Vlan with limited bandwidth / Qos. I imagine the switch are L3 (routing) and that each sub segment has a Vlan ..
Other axis: You should also see if you are in Multicast which is more than possible for 2000 cameras. If your switch is doing slower IGMP request then , it will catch a new multicast stream later ... 2000 camera x 5 Mbit in average (view/record) = 10 Gb just to get stream 1 & 2 without Playback ....
Like for IPTV, it does request very high level switches. (fast query , fast leave, high speed backplane) I have heard directly from Genetec, that even some high end Cisco IGMP swicthes could reach their limits faster than expected on the datasheet because Multicast (viewing) +Unicast (http management & Playback) are using much more CPU and memory than Unicast alone. In that case , it means you will probably have to load balance on several switches to get more power and speed. ..
AW
Alex Wasilesku
Aug 24, 2016Sounds like the interlink between switch communication is going through a smaller pipeline than the rest. I had a very similar problem until I changed out one of my Netgear switch with a top of the line cisco switch, configured mls qos, made sure the interlink between fiber was good, made sure the gbics where performing optimally with my switch etc and it resolved the issue.
U
Undisclosed #3
Aug 24, 2016OK, divide and conquer. First contact the manufacturer of your equipment and get high level support. Have they seen this type of problem before? How did they resolve it? Then what is the latency of the network, fully loaded? It may be a simple slowdown. Do a speed check. Then if that's OK look to shunt (take off line) groups of cameras. Does that fix it? If so look at the group of cameras you removed. Do they or does one camera have a problem? Then reverse any upgrades that may have caused this problem.
In other words dissect the problem down to its smallest part, then you can comprehend what it is and fix it.
rh
ravi hosein
Aug 25, 2016Thank you all...will share every suggestion...tech on...
New discussion
Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.
Newest discussions