Problem With Unreliable 32 Channel NVR - What Should I Do?

my small business has 2 ~10000 sqft buildings separated by storage & parking lots, with a 1" underground conduit between them for cabling. i have an installer who put in a Vitek 32 ch ip nvr (replacing old geovision analog) that is not reliable (video loss, monitor freeze for past 4 months) with only 10 vitek IR cams (3 inside domes, 7 outside bullets) setup on 1 building (other building has no cabling or cams yet) - it continued to fail with a new APC UPS with power conditioning running just the 8 onboard POE cams with all other connections removed (network, monitor). so i'd appreciate any articles you can point me to that i might have missed in the past month of reading, or comments or advice on how to choose the right system. not sure how nonIP systems would work for this setup (cabling?). the (more problematic?) features i'd like include:

all live cams on 1 large monitor ; license plate cameras (have none yet) ; restart recording cams after video loss (which the vitek nvr didnt do even tho cams were streaming fine after whatever glitch caused the loss) ;

i've heard the recording restart called a "watchdog" but was surprised that it wasnt designed into the vitek. what do other manufacturers do? is this not a common feature? will a server & VMS software work better than an nvr? i am a bit price sensitive, but i'll pay for quality & reliabilty. thanks for your assistance


Hello.

You've done a good job a giving details here! If possible, clarify a few things just so the problems can be troubleshot better:

1. Is the underground conduit being used? You mention 10 cameras, but I am unsure if they are using this conduit or all are hung/viewed from the same building.

2. Was the APC UPS providing backup power at all? I would anticipate it worked for a few minutes then dropped dead after the batteries drained. Is this correct?

3. Are all cameras connected to the NVR via cat5/5e/6 cables, or are there coax converters being used?

4. Do you have model numbers on the NVR and Cameras? It is worth checking to make sure the NVR's power budget is sufficient to power all the attached PoE cameras.

Auto-Restarting recording is not uncommon, but I think the goal is to eliminate the power problems to start with.

Assuming you have the NV3200P, the spec on POE power looks sufficient (120W 8 ports) for 802.3af devices on all ports.

...it continued to fail with a new APC UPS with power conditioning running...

I took this as a statement indicating that the mains power is good, is that the case? Or do you suspect a problem there?

This is a problem that actually may be helped by a firmware update if one is available. Because it sounds like it has never worked from the start, and could be explained by the NVR crashing repeatedly.

The Log files are key here as they will hopefully reveal what the failure is when there is video loss and monitor freezing.

Even though the POE power capacity looks adequate, I would ask if the IR cams possibly come on at the same instant, because of a light switch being thrown for instance. There was a test on an AHD DVR where the recorder that was providing power may have been overwhelmed by the instantaneous draw from multiple IR cams, causing it to reset. It's a long shot, but the logs should tell us at least enough to rule this in or out.

Finally, according to the data sheet in the link above, this model does have a watchdog restart process. The fact that it does not seem to be engaging also points towards a possible firmware solution.

Brian -

1. the conduit is 150' and unused as yet - intended for cat6 runs for internet & POE switches

2. new APC ups (model bn1080g, 1080va / 650w, AVR 94-107 +11%, batt output VR 115v +/- 8%) was installed replacing old APC w/o power regulation. connected to building power as normal during weekend test where nvr had its 2 power connections (nvr & POE) & 8 POE cams & nothing else plugged into the nvr or cams. is this not sufficient to isolate a power issue?

3. all cat 6 (red), 5e (blue & green). many cams have new cable where old coax was used as pull line - they go down too (even a 10' new run to cam3 listed below)

4.

VT-NV3200P
www.vitekcctv.com/ProductDetails.asp?ProductID=937

onboard ports 1-8
VTC-IR402
VTC-IR302
VTD-MV20VN
VTC-IR402
VTC-IR402
VTD-MV20VN
VTC-IR302
VTC-IR302

remote POE (4 port)
VTD-MV20VRN
VTC-IR302

no other system at my business shows any power issue - no network or production outages. are nvrs more sensitive to power issues? rather than spend who knows how many hours trying to find something that may not be there, or in the nvr itself, the auto-restart feature should resolve the issue, right or no?

Und2 -

Vitek is working with us to resolve this & updated firmware themselves to the most recent a few weeks ago. the log is sparse, not reporting video loss at all. we talked with the lead developer last week & he said he is working to make logging more robust, and that 'watchdog' restart was not designed into the unit, but he is working with engineering to implement that thru firmware or hardware changes - he didn't know if current hardware would support such a feature.

i dont know if IR startup draw could be THE answer (causing hardware breakdown over time), but maybe some of it. there are flood lights near some cams so the switch to IR should not be all cams at the same time. recording loss seems most often at night, but during the last weekend cam7 was lost @ 1206pm Sat & cams 1,5,6 were lost @ 1108am Sun. according to the recording graphical logs i've been looking at i'd say the unit begins recording again for a cam that was not recording for more than 2 min ~5% of the time.

after many months, i think i'm done problem solving & hope i can find another more robust product or system. does this 2 building layout require an IP system or can others be used? i do want to be able to monitor my business - is 32ch on 1 screen not common or requiring more fancy hardware?

The 32 channel requirement is certainly a limiting factor.

However Hikvision, I mean LTS, released a 32 channel tribrid DVR at ISC West, that on paper sounds good. It has up to 32 channels of TVI/analog input and up to 8 channels of IP, 32 total.

TVI lets you run 2-3x longer (without a device) than standard Ethernet. You can also use existing Category cable with baluns, though your distance may be reduced some. (I know you just ripped out coax.)

32 ch playback is supported as well. My main concern would be on the usability of the monitoring / playback software, as I'm not familiar with it, and have heard some concerns voiced here.

So you could keep 8 of your IP cameras, (need a POE switch) and then the rest 1080p 15fps analog HD.

Amazon has them for around $750, probably a lot less than the Vitek.

Before I would make a recommendation, I would like to ask a few more questions if you don't mind. I like to keep things as simple as possible. I noticed you said you have been at this for many months, so I feel your pain. You are ready to be done with it, but you need to know a bit more about what you have. If it were me, and I have come this far, I would go the rest of the way. I would record just a few cameras at a time. I noticed you did not mention cameras 2, 3, 4 or 8. Is there anything special about where they are mounted, the length or the runs, anything at all? You might want to consider recording just half the cameras at one time. Then the other half. Make sure to use different channels on the NVR. You could easily have issues with particular channels on the NVR. Mix up the cameras and to see if the problems follow the cameras or stay on the channels.

I know Vitek better for their 960 line equipment. No harm intended, but they never impressed me with their technical ability, especially on the IP NVR side of things. Having said that, the cameras actually look pretty good, and they say they are ONVIF compatible. That is important.

As for options, Undisclosed 2 mentions Hikvision and with good reason. They make a good product. I have not seen the new recorder 2 mentions. Just my opinion, Hikvision makes pretty good, basic equipment and the price is excellent. They do lack in tech support, but they are ramping up.

March Networks makes a very good 32 channel machine. I know the company well and their software is mature and has everything an IT guy could ask for and it does support ONVIF cameras. You will like their support, particularly Tier 2. The issue will be price. All that stuff I just talked about cost more money. A good bit more with March; but worth it.

Another option is to install 2 sixteen channel machines. I know some customers prefer 32 channels, but I have never really understood why; to each his/her own. Run 2 machines; Machine 1 has labels 1-16 and Machine 2 has labels 17-32. Your eye will not notice a lot of difference.

Digital Watchdog has 16 channel machines that integrate Vitek camera software. I know DW and you will like their equipment. It is not as competitive as Hikvision or Vitek, but it is still competitive.

Und2 -

from my reading i was understanding that a 8 POE switch (i already have one) in building 2 would help extend the effective/reliable length of an IP cam run. there would be <200' from the switch to the nvr & at most another 100' to any cam there. perhaps if i find a hikvision dealer they can let me visit a site similar to mine to see how it's working there - is that gonna be difficult with hikvision or do they wanna sell me product & show me how well it works?

Mark -

thanks for your comments. there are short runs ~10' & long runs ~100', different cams as you see - the cams you ask about worked for that weekend, but at other times went down - no particular pattern other than cam2 & 7 going down more often (new green cat5e cable run on 7 ~100') & 4 & 5 ~80-100' less often (old cat6 on poles above the storage lot). the strangest part of that test was while cam1 stopped recording & froze on monitor, it was working fine on the webviewer. after doing the failed test with only the 8 POE cams recording with the latest firmware i'm not so interested in doing the testing of various ports & cams you suggest unless vitek /installer offers an adjustment to my expenses to do 'their' testing. the next week will decide that issue.

i'm more willing to give up the 32ch on 1 monitor for a dependable system on 2 monitors (my eye can see an extra monitor quite clearly, lol). the installer who put in the vitek hardware does digital watchdog installs as well & prefers the vitek - i will get more info from him as to why - can you offer more of your perspective about the difference other than price - and anything more about hikvision 16 nvrs?

i will look into March to see what they offer, tho if i return the vitek nvr, the cams will likely go back as well. thanks again for your help.

Based on your comments, I would not do the recommended tests either. You already have, and more. The webviewer test was the clincher. You have narrowed the issues to the NVR itself enough to satisfy even this old dog.

Generally speaking, Vitek is preferred by installers because it is cheaper and allows more profit. Also, their reps are very active, putting the product in front of their customers (installers). Their 960 product line was solid, but Analog is easier to engineer and the results are a known quantity. That technology is easily replicated by a number of manufacturers. IP as you now know is a much different ballgame. It requires R&D and then long term testing that not every manufacturer is willing to invest in. "I's" don't get dotted and "t"s" don'g get crossed. It should not be rushed to the market.

"DW" is manufactured overseas as well, but again, they are a very mature company with a robust R&D department. When they put their name on it, it has been thoroughly tested. Their biggest issue is on the sales side of things. They depend on Rep Firms (some are good and some are bad). Their factory people are really quite good. As for the 2 monitor concept, you can do that but don't have to. They have software that will allow you to display the video from more than one NVR at a time, but I think you are limited to a combination of 16.

March is fully mature product initially designed by and for IT end users. Logs, security, bandwidth throttling are all things they designed into their platforms when IP DVR/NVR's began. In the 13 years I have with them, I have returned no more than 5 machines. Things go wrong yes, but we are able to work with the factory and get them back up. But again, it is clearly more expensive. The feature set is worth it to our Financial customers who have to answer to the FDIC auditors.

Hikvision is a real enigma to most experienced security pros. They have been around for a long time, mostly as an OEM partner, making products for others, so they have the resources. They understand quality and reliability, but keep things simple. In just the last few years they have morphed into a formidable force in the western markets. They do lack a few things, most notably people, but they are ramping that up pretty nicely too. What they are able to do for the prices they charge is baffling. They are either using sub-slave labor or have a heavily automated magic factory somewhere. It would appear on the surface the only thing capable of stopping Hikvision is Hikvision.

You seem pretty savvy to me. Even if you do run across an integrator unwilling to demonstrate products to you without a firm commitment, I think you can handle it. A commitment to a good integrator, even the one you have, will go a long way to solving your problems no matter which manufacturer you select. Just be firm and fair.

Good luck and please visit again. We always enjoy hearing how things turn out. We learn from it.

Edit: The main reason I mention DW is that they include the Vitek camera software in their NVR's. The list of manufacturers that write Vitek code into their recording software that I trust is short, but DW is one. If your cameras are working in the webviewer and you like the results, I would not be so quick to trash them. Money is money.

Is there a reason why you prefer an NVR vs a VMS server? Maybe you are unaware of them? Maybe you had a bad experience? Maybe it seems too complex?

Have you attempted to power the cameras by a standalone PoE Switch? Then you could connect the NVR to the switch. This would alleviate the built in PoE of the NVR as the culprit. If you still had issues, then it would be certainly the NVR.

I, too, attempted to install a Vivotek NVR once. I gave up after a few days of beating my head against a wall. It was in the very early days of IP, so my experience probably isn't much more than anecdotal today. Also, this was before I became a member of IPVM and have the knowledge I do today.

If it were my project, I would seriously give a thought to a VMS based recorder. We commonly use DW Spectrum VMS because of it's simple user interface. If you needed something more feature rich, you could look at some of the other more expensive packages.

Just my 2 cents.

Verify cable integrity with a really good meter ( Fluke ) under load condition (connections) on camera's that are failing.

Make sure all patch cables are same (Cat -6 or better)no old cables

When does this happen ( time of day, night )

What does the camera look at when it fails

is it a system fail or an individual camera fail

is it when full light in motion condition or no activity time of day

Full light or no light

Do you have (extreme lights) High Output )looking at cameras or LED ( Type of light looking at camera

Check details on machine when in full recording mode with all cameras active to see what the Full Load Report Shows ( bits) see if (peaks,Spikes),when failure occurs

Best of luck

Mark - thanks for insights - would you have any comments about a VMS server vs NVR?

Jon - i did ask for advice whether a server is a better option above, so i appreciate your remarks & i've got some reading to do.

I am going to run-a-fowl with most of the members, but I really don't have a preference of VMS over NVR. It depends entirely on the circumstances, and the end user. From reading between the lines, you sound like you can handle a VMS solution. If it were me looking at a VMS, I would probably take a look at a Seneca or Razberi server. They both make a very robust machine, and they have the multiple software platforms built right in. The issue will be camera licenses and support. You will need to purchase licenses for your cameras, and depending upon your choice, clients as well. But if I were to look at a server solution, get one that can handle the load right out of the gate. Mr. Dotson points out that you will have the ability to troubleshoot problems more in-depth with a server and he is correct.

The only problem you really have, and I hate to say it so bluntly, is that you were sold an NVR that is in its' infancy. There are plenty of units out there that can service your needs quite well that have a good history of performance. Salient Systems, Exaq, March Networks, are just a few of the mid-priced providers that have an excellent reputation.

A Mark mentioned...take a close look at the NVR itself.

It was mentioned that you has 32 streams all being shown.

That is a huge load for any CPU no matter what OS it is.
If the OS lets you look at the real CPU utilization..this will help a lot.

The few embededd linux units I have looked at were not very good at allowing users to look at the real system resources like we can in Windows based ones.

I hate to hear that any end user has been sold a bad deal from our industry , but it happens. If you were in the Virginia area I could and would be of help to you sir.

Again my apologies from a professional Video systems installation company.

Seems reducing the camera counts from 10 to 5 would have been a decent test.

Are you using the same LAN as desktops as the transport for these cameras?

are you using VoIP phones?

I'm more inclined to believe you have a LAN transport issue,

Dumb, smart or fully managed L2 / L3 switches....

I've seen all these cause issues....

I'd want to know the exact cause of the current problem, before you consider investing in another platform...

IP isn't that difficult to troubleshoot, you simply have things in some order to properly isolate what is from what isn't working......

He has an NVR with a built in PoE switch. There likely aren't any other hosts on that subnet.

I'd also never consider placing Copper between these two buildings. You can buy a 200' or so length of indoor/outdoor pre-terminated 6 fiber (single-mode) and enclosures and an SC fiber panels for around $400-$450. Avoids a lot of limitations you would face otherwise....

IE - With Fiber, many Switch makers will allow you to STACK the switches and manage them as a single switch.....and have a 10G or more backplane between the switches.

The question that wasn't asked, is prior to the beginning of the failures four months ago, did the new system work properly for any period after it replaced the analog system?

Jon is right, and use the fact that you can isolate the DVR and Cameras from everything else, and built the system up with the help of the vendor for the proper firmware and a default factory reset, and pre-made patch cords to built the system on a test bench one camera at a time. This should clearly expose where the problem is.

We keep a lot of 290' patch cords on those nice orange A/C wind up reels in the office for testing. We terminate both ends on CAT5/6E blocks and use patch cords to connect, duplicating horizontal runs....

You should be able to isolate the issue very quickly.....

Andrew brings up a good point in that the quality of your POE DSE is important in this case.

In my lab I have a ByteBrothers POE1000IL that I use when testing our hybrid POE servers.

You can look at the source end and the destination end to see if the supplied voltage is within POE spec. Below are the values from the wikipedia page....af and at versions.

To test...turn on all the IR on cams that have them then monitor the voltages.

If any cams are PTZ...put them in a 'patrol' mode so they are drawing more power.

Power available at PD[note 1] 12.95 W 25.50 W
Maximum power delivered by PSE 15.40 W 30.0 W
Voltage range (at PSE) 44.0–57.0 V[20] 50.0–57.0 V[20]
Voltage range (at PD) 37.0–57.0 V[21]

42.5–57.0 V[21]

my thanks to all for their informative posts - so far Vitek is not providing an incentive to continue testing their NVR which has failed to provide what was promised, tho I am considering keeping these cams with any new NVR or VMS server.

i'd appreciate any comments about what to consider or look out for regarding mixing cams with NVRs or VMSs with regards to licensing, etc. i am going to talk with some vendors/integrators to get a better idea of my options & costs.

Full disclosure - I work for Digital Watchdog

Jon and Mark thanks for the kind words, appreciate the support.

EU1 If you would like to evaluate the VMS recommended above please feel free to reach out to me directly at patrick@digital-watchdog.com. I can get you lined up with a local resource.

Thanks

Patrick