Luke, thanks for sharing! Great idea!
We'll figure out a prize and some recognition for the first (non-Dearing :) person to solve this.
NTSC vs PAL settings got changed?
DVR did not like the square wave output generated by the cheap UPS when the power went off
The Modem and patch cable are good because you connected and stayed connected - Check.
The DVR showed previous playback - Video output is good - Check.
Still have video on the hard drives - Hard drives good - Check.
All cameras showed the same symptoms at the same time - Two things that are common to all cameras at the same time are power and video input card (Unlikely that cable all went bad at the same time).
Unlikely that all inputs died at the same time. Individual cameras dying would indicate a single video input lost on a card. All dying at once, with the same symptoms would indicate your power supply probably blew a fuse or tripped an internal breaker, depending on what kind it is. Power is common to all of your cameras.
- My 1st. guess: since all BNC ports on the DVR's side combine and end in one single daughterboard circuitry, a surge in voltage running through the wire from any camera to DVR (storm and heavy rainfall; lightning strike will find it's shortest way to ground through any piece of metal in the ceilling or conductor wires that are in it's path) simply messed up with the channels and it just "went completely crazy" and that "crazyness" showed on-screen in the monitor. Signals meant for channel 1 showed up in channel 8 and viceversa. And then the first or even a second voltage spike fried either the camera or the daughterboard.
- You didn't say if coaxial cable end-to-end, or ethernet Cat.5/6 cables with video baluns were used in this case. If it is ethernet network cable with video baluns, consider yourself "luckier" if the VIDEO BALUNS sacrificed themselves ($12 dollars or less a pair, way much cheaper to replace than any DVR or camera) by taking the surge strike themselves and protecting the more expensive components by frying it's own (meaning the BALUN´s) internal circuitry.
- The camera view from the video you uploaded seems to be an indoor scene. If the camera's bracket was drilled directly to the wall inside the house, the wall's material typically will either conduct electricity very very poorly or none at all. You didn't say if the camera is installed touching or near a metal beam. If that is the case, then my guess is the voltage surge used the beam as a path to the camera; and continued to the DVR.
- My 2nd. guess: the customer simply did "something", which could be anything you can imagine. (In my years of experience in the field, many cases involve finally the end user themselves). They touched or clicked or disconnected something they were not supposed to, and when they messed up and cannot undo it, they call you to say "it just happened and now it is like that" : (.
P.S. By the way; I've seen that more than 90% of customers have the perception that simply adding a UPS battery will solve all their power backup (in case of a power outage blackout) and surge supression (in case of a hight voltage spike) problems. But maybe (partly because of the great marketing job some manufacturers are doing, partly because we integrators ourselves simply throw-in a UPS to the project for this purpose without further thought) that solution will only give them a sense of false protection / flawed peace of mind, because if the UPS doesn't cut off the voltage spike NOT ONLY COMPLETELY but also AUTOMATICALLY 100% of the time, or burne itself, or burned it's integrated fuse, then the UPS is simply an expensive 6 outlets power strip providing a couple minutes of extra power.
One big clue must be the time stamp which goes from :13 seconds for the opening scene on View 'A' to :19 seconds for View 'B' and then continuing to oscillate between forward and backward time stamps. Since we assume these CVI cameras don't keep time individually but rather only one time is kept on the DVR, this time-traveling interleaving seems perplexing.
However any good mystery watcher knows that seemingly unimportant facts like 'the ADSL modem not being connected to the UPS' are key. Therefore that plus the fact that it seems well nigh impossible for a sequential stream to have frames anything but appended, no matter what the source, indicates that the NTP connection was possibly syncing and freewheeling as the connection to the Internet was lost. Perhaps in the waning moments of power, the video threshold voltage dipped and resurged causing the cameras to re-order themselves in their dying gasps?
Actually I have no idea.... :)
Luke one question is one about how specific your solution is, i.e. does your explanation account 100% for all weird behavior, without fail? Could you replicate this failure at will?
Can you post another camera's clip before going black? Also when you say the Samsung monitor was black and the views were black, do you mean the monitor was off? Do these cameras keep their own time and create their own overlays? I would think not, but who knows?
I don't know but I'm hoping it involves bears.
Are you using an offline UPS?
DVR and CVI cameras were out of sync when the power outage happened. (swithed to battery power with inverter)
Time stamp and camera overlay continues to work(this overly is not coming from the camera), therefore the Hisilicon DVR chip was encoding and storing to disk properly. The reason you are seeing multiple cameras on "one" channel is that the video input to the Hisilicon DVR chip is getting confused - it gets multiple interleaved channels from the HDCVI video decoders into its video port(s). The DVR chip is intermittantly losing synchronization. So now we've isolated the problem to the HDCVI decoder chips. So either they have gone on "walk about", or something from all the cameras have caused all the decoder chips to lose their minds.
The analog portion of the DVR (the video decoder front ends, which connect to the cameras) should have a common ground connecting the DVR and cameras (but they should have some ESD/Lightning Strike protection diodes). Additionally, the analog portion of the video decoders will have a common power and ground plane that is DIFFERENT than the digital power/ground planes that the Hisilicion DVR chip uses.
So my supposition is either a) lightning strike messed up the common ground between cameras/DVR or b) power spike affected only the analog power/gnd portion of the DVR, causing the HDCVI video decoders to stop working.
Either way, the fix would be a simple reboot of the DVR and cameras (assuming nothing was damaged). If, however, the analog power/gnd portion of the DVR got toasted, you probably replaced the DVR?
It is *possible* that all eight cameras got damaged, but I don't think so.
The DVR chip will have only one clock. It will place the same timestamp on all channels if it is turned on. It can be turned off if each camera is providing its own timestamp - but HDCVI probably doesn't have time metadata in their analog stream yet, so you set up the DVR to provide the camera number and the timestamp.
From what I can see in the video, the overlay is not from the camera, it is generated by the DVR chip. I see the timestamp change from bright to dark (called Alpha-blending) depend on the background, but I don't see the actual timestamp jump around - it counts forward from 21:49:13 to 21:49:28, with maybe some jitter to it (could be how Youtube plays it or how it was uploaded).
The label "Cam 8" is also probably generated by the DVR chip - labelled when setup accured. This stays consistent regardless of what scene is shown, telling me the video is getting mixed up prior to the DVR chip applying the timestamp. The timestamp typically happens after the video is captured, but before it is compressed using H.264.
So I still think something is wrong in the HDCVI decoder chips and their sync/connection with the DVR chip or the cameras.
Of course, I could be wrong...
Hey Luke, can you confirm or deny Robert's and B's lightning theory as the cause of the blackness, as well as Robert's cold boot as the solution?
They both seem most plausible, but if they were ruled out then that might inspire a new round of wildly speculative and highly entertaining scenarios. ;)
The possible scenario may be as follows
(a) It appears that the power source for camears and DVR are different. Thus the power supply was intact for DVR and the power supply for cameras might have got disrupted
(b) The breaking of the images may be due to short circuit and the time intervening between tripping of MCB and fluctuation in the voltage.
(c) Another possible cause may be due to lightening strike the Power source for Cameras.
As brought above I feel the black out of cameras may be due to complete disruption of Power supply.
My guess is that the power went out, the system then kicked onto UPS only. The UPS voltage then went low scrambling the cameras. Cameras kicked off first (IR uses more juice). Then the monitor was using minimal power (only displaying the minimal DVR frame and black leaving even more battery power for the DVR to hang on (which was using nearly nothing as processors are idling). Then possibly the DVR was powering the modem through the network port (unless modem was on another circuit that stayed alive and it was somehow powering the DVR? or the DVR itself is POE?) Solution: hit breaker which tripped in storm or wait till power company crawls out of bed). It seems in the video there are no devices powered on in the house (no appliance LEDs lit up in frame that I could see anyway).
I posted this a half hour ago or so I thought.
My guess is the video card. My experience when we used DVR's was whenever I heard or saw "camera bleed", (Camera 3 showing up or blinking when viewing camera 1) it always turned out to be a video card issue. The old Integral units I used this happened when the card wasn't seated properly or was about to fail. Even happened on new units the card would get jolted in shipping.
If that's not the case my second guess would be some strange power issue with that brand cameras.
IPVMU Certified | 05/21/15 09:38pm
I am going to vote for a short in the monitor or associated cable. I have experienced this before and replacing one or the other solved the problem. Disconnecting the monitor from the recorder restored remote images to their proper display.
I have also seen the recorder itself take a surge and exhibit similar symptoms and the recorder had to be replaced.
My guess would be water leaking into a camera connection and shorting the 12vdc to a video input - fixing meant disconnecting inputs one at a time until shorted input lifted.
IPVMU Certified | 05/22/15 03:07am
Firstly, thank you to everyone who joined in on this whodunnit. It has been fascinating to read so many different approaches to solving the same problem. Some of the responses were quite detailed and a lot of thought went into them. While I posted this just as a fun challenge, I've also learnt a lot from the many knowledgable contributors, especially about the inner workings of a DVR to explain why the video went so weird.
And the winner is … Mark Jones! More about Mark’s answer in a moment.
What was the cause of the problem?
The cameras were all supplied power by a single AC/DC power adapter which was connected to a UPS. The AC/DC power adapter died with only a few seconds of warning so all the cameras stopped working. The green LED on the power adapter was no longer lit so this made it easier to confirm the fault.
The DVR had its own power supply so it kept working and simply recorded black video.
The monitor had its own power supply and was working despite the customer saying that it had gone black and wasn’t working.
What was the solution?
The solution was simply to replace the inexpensive AC/DC power adapter.
It’s difficult to say why the power adapter died. It cannot have helped that the electrician, who originally wired power to the cameras, mistakenly thought they used AC power. There is a green LED on the power adapter and it switched off when camera power connections were connected with the wrong polarity. It’s possible this shortened the life of the power adapter although I was more worried about the impact on the cameras at the time! It was this same LED which helped confirm that the power adapter was dead.
Not only was Mark Jones the first member to identify the failure of the power supply to the cameras as the likely cause of the problem but Mark’s detective work was elegant in its logic and simplicity. He identified the components making up the CCTV system, and systematically eliminated all that he could from being a cause of the problem. He narrowed this down to one likely candidate and he was right. Congratulations Mark!
Challenges in solving the problem
In retrospect the problem and solution are simple enough but there were a few challenges to finding and confirming the problem.
- The customer site was remote from me so I could not just drop in, check out the equipment and find the problem myself.
- The customer was non-technical so my troubleshooting questions needed to be straightforward with no jargon.
- The customer did not speak the same language. I suspect the reference to the monitor going black may not have been an accurate translation. While I always take note of what customers tell me, I also try to find ways to verify the information they give me without causing offence! In this case, I was able to use Skype video to see the DVR interface showing on the attached monitor but all camera feeds were black.
- It was unclear if lightning was involved in affecting the CCTV system as the customer was asleep at the time. I checked the camera recordings for the 20 minutes before the camera blackout and found no evidence of illumination by lightning so I ruled it out as a probable cause.
- I also wondered whether there had been a general power outage but as the customer had been asleep, no outage was noticed. By taking remote control of a computer at the customer’s site, I was able to access the admin interface to the customer’s ADSL router/modem and check it’s “up time”. The admin interface showed the modem had been connected since well before the camera blackout event and was still connected. Given the ADSL router/modem had been working the whole time even though it was not protected by the UPS, I ruled out a power outage or power spike as a probable cause.
By ruling out some nasty scenarios, it quickly seemed likely that the cause was limited to internal problems and not caused by outside influences. Asking the customer to check the green LED on the cameras’ power supply rapidly confirmed the simple cause of the problem.
I get a kick out of solving technical problems when they happen and I hope you’ve found this fun too!
Wow, eight cameras on one power supply. Maybe one like this?
Congrats Mark! Thanks Luke!
Thank's Luke, it was a fun exercise. To be honest, I did not notice that the first post was a video. I thought it was just a picture. Once I watched it I began to have a few doubts but decided not to let electrical goblins change my mind. As my reward, I will post one back. We had a good one recently.
I was just reading yesterday on ways to make physical security a more fun topic. There is very little out there. Everyone is so serious! This "tech-challenge" is a good idea Luke.
IPVMU Certified | 05/24/15 06:40am
what is the power supply Ampere ?
and this problem on morning only or morning and night?
Mohammed, It was only first noticed last Thursday morning. Given the nature of the actual problem, it was morning and night. Other doors locks from the same power supply functioned normally.
By the way, just thinking about your question of splitting the HD-CVI camera power load over two power supplies, would you use two 4-port PoE switches to power 8 IP cameras or would you just use a single 8-port PoE switch?
Luke, I wouldn't have said anything except that you said you were going to have TWO power supply units on-site, a working one and a spare, and you had not posted the camera models nor their power usage yet, so its only natural to consider using both, even if under the max power draw of one.
Still if I had a power supply that was running at 'almost' 70% capacity, and I had another in a box at the site, I would certainly consider instead running them both at 35% to extend their life. Also, if one died you would still have 4 cameras that would continue to function, instead of the whole system being down until the user notices and performs the swap out.
In the case of dual-use power, instead of swapping out an old unit, the user would just transfer the 4 from the one to the other while waiting for a replacement.
Of course they both could go at the same time (lightning, surge), but then one would expect to have other failures as well (DVR power, UPS) that would prevent the system from functioning at all. I'm not saying its defintely the way to go, but its worth talking about right?
One other thing about the power Luke, how long are those eight 12v runs? And what gauge wire are you running them on?
Thanks for making me think with your insidious challenge!