Unstable Camera Sync Or Flaky Encoders?

We are having issues with two-channel IndigoVision 9000E encoders, particularly in conjunction with Pelco C10CH-6 cameras. The issue is that when the cameras temporarily lose power or their power is provided by backup generators, some encoder channels will develop a color bar at the bottom of the screen, often accompanied by a massive bit rate increase in recordings. Oddly enough, this sometimes affects only one of the three streams the encoder provides: one TCP/IP Unicast for recording; one UDP Unicast for certain viewers and one Multicast for other viewers (I know it's complex but IV's encoder chassis are only capable of 40Mbps total Multicast streams while allowing up to eight simultaneous Unicast streams per channel (320Mbps max) and with 20 channels per chassis, the decision was made to feed some users Unicast for "Live" while feeding other users Multicast).

Anyway, the Integrator contends our problem is caused by poor wiring or poor power and wants us to provide detailed "camera #, location, type, cable type, cable runs for video (termination point), power cabling, electrical power termination, electrical phase, and grounding plane" to assist in troubleshooting. I contend that since more than 90% of the problems happen on channels that are connected to Pelco C10CH-6 cameras and since we are in the middle of replacing them anyway due to their absolutely awful picture quality (we regretted buying them in the first place but it was/is difficult to find cameras small enough to fit in Pelco DF5 back boxes), we should complete the camera replacement plan and then see if the encoder problems continue before doing literally weeks of "unnecessary" work.

I also believe there is something peculiar in the design of the IndigoVision encoders that makes them susceptible to instable sync - the likely primary trigger of the problem. In essence, I believe that the C10CH-6 cameras exhibit some issue with video sync on startup and I also believe the IndigoVision encoders are particularly sensitive to that unstable sync.

Additional info:

  • Pelco C10CH-6 cameras are a relatively and increasingly small percentage of our total analog camera count. We probably have less than 100 left compared to hundreds of inMotion box cameras; more hundreds of Vitek dome cameras and a mishmash of other cameras.
  • We mostly see the problem happen with the Pelco C10's. Only one or two channels may have had other camera brands/models connected to them at the time they exhibited the problem.
  • One oddity is that sometimes only one stream is affected while the other streams exhibit normal behavior. This also makes it tough to troubleshoot because basically we have to check each input three times; once for each stream. We have to run through all 900+ channels in "Live" on a secondary client (uses UDP Unicast), then run through them again on our Monitor Wall (uses Multicast), then check the recordings for high bit rates (channels that record the color bars also typically run at inordinately high bit rates - as high as 15-25Mbps versus normal of 2.5-3.0Mbps).

The big questions: Does anyone have any other ideas? Does anyone agree with the Integrator?


"Oddly enough, this sometimes affects only one of the three streams the encoder provides"

This makes me think the primary issue is with the encoders as if it was simply a camera problem every stream should be impacted.

What was IndigoVision's response to the situation?

John,

I think the problem is two-fold: The Pelco C10CH-6 cameras are highly suspect due to their being the the ones whose channels exhibit well over 90% of the problems. They are also an ever-decreasing percentage of our overall analog camera count (probably not much over 10%) as we had already started replacing them due to their video showing jagged edges and diagonal "bands" of color even on our old analog matrix. That was apparent even when the cameras were new.

I do agree that a secondary issue may be some incompatibility between the encoders and something that happens within the cameras during power cycle or while running on generator with the cameras in "line lock" mode.

The part that does bug me is that this doesn't always affect all encoder streams. Recording is via TCP/IP from Encoder 1 while "live" viewing is either by a UDP stream or a multicast stream from either Encoder 1 or Encoder 2, depending on the user group viewing the cameras.

I would hope that replacement of all of the C10CH-6 cameras (and additionally switching our remaining other brands and models to "Internal Sync") either resolves or at least drastically reduces the occurrence of the problem but replacing approximately 100+ cameras and switching hundreds more is not a simple task.

By the way, both IV and the Integrator were investigating the problem at their end but I told them they could put it on the back burner unless it continues after all of the C10's have been replaced.

On the other hand, I absolutely refuse to believe the problems are caused by bad wiring, improper termination, faulty power supplies (after all, what ever goes wrong with Altronix 24VAC supplies?) or power line phasing. And it would be nearly impossible to rewire the entire camera power subsystem, including circuits and breakers anyway. Most of the supplies are on the same power phase anyway; something we insisted on when the building was constructed in 2000.

And I don't blame IndigoVision. These issues may or may not be related to issues with our old Honeywell Enterprise system, whose encoders would occasionally start dropping frames on random channels after a power failure or generator test. In that case, we didn't document which channels were affected and it affected both Multiicast "live" and Unicast recording equally. In addition, the problem was much easier to track down since the recording servers displayed total frame rate. Dividing that by the number of inputs attached would come up with 30fps. Any significant variance from that figure would point to one or more channels dropping frames and it was a simple task to check the frame rate on each of the up to 32 channels per server.

IV is much more difficult to troubleshhoot since we are recording many more channels per server (up to 68) and the fact that the problem doesn't always affect all streams. Actually, we're recording more channels than that per server because each server runs multiple instances of their software.

Carl, it does seem to be statiscally incriminating that 90% of the problem occurs on ~10% (?) of the hardware. I'm curious if any of the analog runs between cameras and encoders have been shown to be more or less susceptible than others. You I assume have runs of differing lengths and cable age even if nothing else. One would think that an interment 'boundary' type condition would be more likely to manifest on the certain, i.e. weaker parts of the system, more often, giving possible clues. The one plus of having so many cameras installed, is that it should make it easier to swap and isloate components.

A few questions:

You say the problem occurs after power failure, how do you fix it at that point? (Cycle the indigos?)

Roughly what are the 'odds' that a given Pelco would exhibit this behavior if cycled once? What about the other cameras?

Are there any parts of the infrastructure (besides the Pelco cameras themselves) that are unusually correlated with the problem?

You say that it is troubling that it doesn't always happen on all the streams at the same time, 2 of the streams are would be at a different resolution I believe, does that show any correlation to the problem, i.e. always shows on high-res stream(s)?

Rukmini,

We've tried cycling power at the camera by disconnecting/reconnecting a power wire. We were able to reproduce the issue one time, with one C10 camera. Successive attempts to re-duplicate the problem at the same camera were not successful. Attempts to duplicate the problem with a different C10 camera whose encoder had also exhibited the problem during a power outage were also unsuccesful. in our casino, camera power is not UPS'd (except fixed IP cameras running on PoE - the switches are on UPS power) but our entire Server Room, including all encoders, is.

Yes, the solution is to soft reboot the encoder(s).

Affected cameras are located in different areas, on different power supplies, and the video is transported via different means. Some signals of affected cameras are transported via coax, some via passive-passive UTP and at least two affected cameras via passive-active UTP. Also, at least one camera of another brand and type (a Ganz ZCD-3094NHAT dome) was affected at some time. That is my concern - while the vast majority were Pelco C10CH-6 cameras, at least one or two were different brands and models.

IV also provided us a batch tool to force all encoder inputs to NTSC. They come from the factory in "Auto", which detects the input signal and switches the encoder to NTSC or PAL. After the batch file was run, another power interruption caused five encoders to "hiccup".

To continue, I believe all affected cameras were set to "line lock", including the Ganz dome. None of our most-recently installed cameras - inMotion in11S3N2D, 11S4N2D and Vitek VTD-A2812 were affected. The inMotion cameras do not have line lock capability and, as of ~ a year ago, we have been setting all cameras that have that capability to "internal sync" because we have eliminated our analog matrix - the primary reason for choosing line lock in the first place.

Line lock is preferred when using matrix switching to minimize picture rolling during camera-to-monitor changes. But it did give wavering images when the cameras were on generator power due to generators not being able to maintain constant line phase.

Actually, all streams are set to the same encode settings, including resolution, bit rate and GOP values. That was done by the Integrator and is something I have looked at changing but haven't gotten around to it.

...encoder channels will develop a color bar at the bottom of the screen, often accompanied by a massive bit rate increase in recordings.

Sorry no answers, just other questions:

What exactly do you mean by color bar, not a SMPTE color bar right? Monochrome? How big? Is the rest of the picture unaffected? Is the color bar modulating/noisy?, otherwise how do you explain the bit rate increase?

Just to confirm: You don't have any analog monitoring, correct? So there is no reason for line lock/frame sychroization anymore true/false?

No, not an SMPTE color bar but a horizontal band of alternating colors covering maybe the bottom 10% of the screen. The colors do move horizontally. Sorry, I can't post a picture due to regulations.

Yes, correct. We no longer have analog monitoring. We are in the process of replacing some cameras and switching the rest to internal sync but with over 800 fixed analog cameras, it will take quite a while.

Forgive me if you have already answered any of these above but:

1. Have you reconfigured any Pelcos to use internal sync?

2. If so, have you had any of those manifest any symptoms since then?

3. If not reconfigured yet, is it because you can't do it thru the OSD? OScope work required? (God forbid).

4. When you get rid of the Pelcos, thereby fixing 90%, and so occasionally still have the problem, do you care?

Here's my best WAG at the moment:(assuming line lock=true)

Premises: (indicate if disagree)

1. Encoders do not affect cameras thru the video feed, assuming proper termination. (RS-485 aside).

2. Cameras do affect encoders thru video feed.

3. Encoders must sample incoming video signal to extract sync signal and then slave to it.

4. When power is cycled to cameras, sync is lost at encoder.

5. As camera boots, camera must sample mains to find phase and then slave to it.

6. As camera slaves to mains, sync is generated once again by camera.

7. Encoder must resample video and slave to it.

8. When problem occurs, cycling encoder fixes it.

Speculation:

Whos fault is it?

The fact that you can cycle the indigo to fix the problem, coupled with the fact that such cycling does not affect the camera, leads one to the conclusion that the indigo is not properly slaving to the camera and so is primarily at fault. (Think of it this way: whatever it does when it recycles, it should just do without recycling!)

Why do (mostly) the Pelcos manifest?

Perhaps the Pelcos as you imply do not handle the sampling of an erratic main gracefully and end up sending erractically pulsed video which causes the encoder error when sampling. Maybe the other cameras because they are DC or simply designed better handle the sync generation more gracefully...

What about the wiring?

Seems highly unlikely in any case because unlike the other two suspects it is virtually invariant, it goes thru no state change. Furthermore the problem is operating across a wide variety of various balanced and un-balanced media without showing any correlation to the effect.

Rukmini,

That pretty much jibes with my thinking. Although we could switch the C10's to line lock, it must be done at the camera. The joystick that controls the OSD is located on the side and difficult to reach within a DF5 back box so we would have to remove the camera from its mount to adjust it. At that point, we might as well just replace it.

The biggest issues we have are the intermittency of the problem (it never affects more than a few of even the C10's on any given power event), the fact that it has affected other cameras, the cost to replace otherwise acceptable cameras (C10 cameras could be used for less-critical applications), the time involved to replace or reconfigure hundreds of cameras, the constant need to cycle through every analog camera three times daily (once for each stream) and the fact that we don't understand, nor are we able to reliably duplicate the problem.

Reconfiguring the C10's is an option but since we cannot duplicate the problem on our bench, we may never know if that resolves it, at least unless we reconfigure all of them and wait until the next power failure or possibly at least until the next generator test to see if that is a solution. When a problem strikes only a few of many cameras randomly, how do you track it down?

Do I care? Of course! Even if replacing the C10's resolves 90% of the problems, that still leaves us with an unknown that could, at some point, bite us. Then it just offends my sensibilities. I absolutely hate unresolved technical mysteries. It will continue to be a splinter under my fingernail indefinitely.