Mohammed, sorry to hear about that, that is a tough one.
Can you try connecting one or two of those Arecont cameras to another VMS (like Exacq or Milestone)? My rationale for this is that if you connect it to those and you do NOT have skipping / freezing, than you care confidently go to IndigoVision and have them look into potential issues on their end. Alternatively, if you still have skipping / freezing on another VMS, then it's pretty clear an Arecont issue.
Good luck though, Arecont's infamous for their poor customer service and IndigoVision wants you to use their own cameras ;)
Inaxsys Security Systems | 06/07/14 04:32pm
Are all of these cameras connected to a single server? How many frames per second?
Each of those AV8365 cameras has 4 x 2MP sensors so it is like having 96 x 2MP cameras plus 16 x 1MP cameras. If this is on a single server, this seems like a significant amount of bandwidth/throughput...
I do agree: check out VMS bandwitdh max input and outputs... ( not only inputs) because video wall and any client outputs using basic high resolution streams could saturate the server CPU as well. Multicast settings as well (does explain why we can't put too many camera per server with Genetec)
That's why also VMS using second or third low streams are recommended instead
then max Read / Write in term of bandwitdth trhoughput on the NAS, not only fps writting
Keep in mind a 8 cameras synchronized reading in fast speed can take 8x 30 Mbits= 240 mbits ! only for one operator working on a search.
VMS Bandwith Input: (but also throughtput to the NAS) so use same for NAS output
8 Megapixels : 24x8*=Mpixels outdoor with moves & lights changes= 192 for the REC only ...
10 Mpixels cameras : 16x10* Mpixels outdoor with moves & lights changes= +/- 160Mb
(*may be more at night) TOTAL: +/- 352 Mb/s input and 352 Mb/s output to NAS
If your server is doing motion PTZ tours, motion detect, or analytic add some CPU, used by pre/post alarms and analytic engines
+ any optionnal display streams : for example, VGA for multiple displays or QVGA for SmartPhones
So CPU should stand 700 Mb instant treatment minimum.. not easy on a single server
And during replay .. could reach 940 Mb ....(in that example)
Port miroring feature on Level2+ switchs can help to see whats coming in and out from the VMS during normal usage. Night time is always the best to do the stress test, with synchronized replay which probably kill most systems
I had this issue with a set of three avigilon cams running in JPEG2000 and one LG tied onto avigilon VMS running with a small four port 10/100 switch. I ended up swapping over to H-264 on all of the channels and bandwidth dropped significantly from say double digit (10-14)+ Mb per channel to 2-4 Mb per channel if I remember correctly. As soon as we did that the frame freeze loss issue disappeared. Our conclusion... bandwidth across the little 10-100 switch was too high for the location. We think the switch might be to blame as we are only running four channels on the recording unit.
IPVMU Certified | 06/07/14 11:55pm
Hi Mohammed, is this a new problem or have dropped frames always occurred? If it is a new problem, what has changed since it was working reliably, e.g. added more cameras, replaced an old NVR with a new model?
IndigoVision Ltd | IPVMU Certified | 06/08/14 01:40am
First thing that would come to mind is check the Tput
From the datasheet (http://www.indigovision.com/documents/public/datasheets/Enterprise-NVR-AS-4000_Datasheet_A4.pdf):
100 channel version
Performance (Streams) 100 Recording plus simultaneous playback of 25 streams
Performance (Throughput) 128 Mbps in; 200 Mbps out; 328 Mbps total
200 channel version
Performance (Streams) 200 Recording plus simultaneous playback of 25 streams
Performance (Throughput) 500 Mbps in; 200 Mbps out; 700 Mbps total
Second thing that would come to mind is check the Network.
Are you using Multicast? If you network multicast is not configured well, you would probably get a disconnect from the cameras every 30sec.
Run Wireshark to see if there are any packets drops. In wireshark go to Telephony > RTP > Show All Streams to analyze each camera stream.
I would recommed you to contact your local partner support team - If this is a NVR issue we would be happy to assist you to resolve this matter and escelate - you can send a mail with all the relevant info to the global partner support mail - firstname.lastname@example.org -
Ricardo gave the solution : your 100 channels Indigo is probably saturated (even a 200 would be during searches and replay if you have a video wall)
You know that with an excel sheet. Don't spend time on network, but you can checkout NVR real bandwidth with the port miroring on the switch, easier than Wireshark. It s an IT procedure, no blabla. I'm very surprised people don't use it.
128 Mbps in; 200 Mbps out; 328 Mbps total... very low Input and completly impossible to fast speed replay 8 Arencont cameras (+/- 240 Mbits..) ! no power margin for that
Current CPU is overloaded and probably reach 70 to 100% all the time.
=> If you want to keep your NVR, you need to tradeoff: reduce fps, and quality, use second stream resolution for display, use CBR locked in, based on what cameras delivers in worst conditions in VBR (at night ) to keep a suffisient quality. If the Indigo pushes its own settings to the cameras (check that out) , like VBR and quality presets your are done. Then you should change for a more powerfull nvr.
If you can connect correctly from a new independant PC , to the cameras usually freezing with VLC and checkout that the Fps and bandwidth match what you expect, it does prove: the cameras are not saturated, the network is working well, BUT your server and/or your clients are overloaded. (could be Indigo and milestone, as with 40 very high megapixel cameras... it's like with 80 HD/FHD....
Why isn't someone from Arecont responding here? I haven't dealt with them directly in several years but it would seem that they are still their own biggest enemy. If you call and complain, experience tells me they will eventually give you your money back...and then tell you to never buy their cameras again. Just saying.
Test the live feed of the cameras through the web browser, and keep it running for a few hours. Compare the quality of the video stream (browser VS NVR) and determine if your NVR is the cause or is it the network/camera.
Same quality on the NVR and the Browser:
If you’re getting the same quality on both, it wouldn’t be an NVR issue. Try reducing the camera count on edge switch (one a time) until you get the quality you’re looking for. Consider upgrading the edge switch.
In case the quality remains even after reducing the load on the edge switch, call Erecont.
Better quality on the Browser:
The NVR can’t take the load, or the camera’s driver on the NVR needs some work.
- You might want to connect one of the cameras directly to the NVR (no switches) and check if you still get the same results, that could tell you if you have a driver issue.
- If these cameras are out door, check if the quality is dropping around mid-day, might be the heat.
- What switch models are you using? How many cameras per switch? Is the POE consistent?
- What is the cables length? What is the quality of the cable?
gave a pretty good breakdown of how to analyze the problem, but I have another question..... Are there any non-Arecont cameras also attached, and if so, what brand are they?
Good News is that Arecont's Technical team immediately on this issue from their side and inspecting all other possibilities . Thanks for the same.
Yes, but you should keep in mind that most issues are not coming from the camera ...(only if they are saturated with toomany streams and motion detection..) Most guys from vendors doens't know how a VMS is working and how to test a CPU load. I don't know what Arecont team can suggest on this ...just update your firmware ?
I am curious how this worked out. I have heard of this with other Arecont Cameras (not on Indigo or Exaq) And yeah, where is the vendor in this conversation? I am sort of assuming the awesome staff at ipvm.com pinged them for a response. Hope the project was able to deal.