Subscriber Discussion

How To Prevent Network Flood Attack On NVR?

UI
Undisclosed Integrator #1
Aug 22, 2015

In a period of several weeks, we've received many service calls from the end user complaining about the IP cameras losing video in the early morning hours between 12 midnight - 2 am. The IT manager says the NVR had to be reset because: although the Windows 7 OS is up and running, the VMS doesn't show up any live cameras on-screen and had to be hard-reset to pull up the cameras again automatically.

Upon on-site audit: by checking the videos of the IT room, no one was seen entering this room at thouse hours. In the end, the security integrator was told by us to take immediate measures so as to discard the possibility of blackouts or undervoltages in the 110V electrical supply that no one is aware of during those night hours; we instructed him to connect the USB-Data Port cable to feed the Powerchute software that can then log any electric power incident.

Having said that, we're still highly suspicious of possible rogue behavior from internal IT personnel.

- Is it possible that some unauthorized employee on the LAN is "flooding" the NVRs network card (or IP address for that matter) with excessive requests to cause a denial of service from the NVR ?? (since the NVR has a private IP address from the LAN, it may be easily discoverable using an IP scanner and anyone can easily see that the web server is running on port 80). We also think that if the IT employees have remote access back to the company's network, they may very well do an attack remotely from home, an Internet cafe or elsewhere.

- If yes; can a hardware-based firewall prevent or at least minimize these incidents of internal flood attacks (as in putting a firewall between the NVR's PoE switch and the rest of the IT network and employees PC workstations to restrict incoming requests) ?? I've been told by a colleague that Windows' OS firewall is essentially "useless" because if the OS is unstable, the software firewall comes crashing down along with the operating system.

- We were thinking first of completely isolating the security LAN equipments from the rest of the IT networks, but that would represent other political and day-to-day workflow challenges (the security boss' workstation is on the same LAN as the rest of the employees and has to be able to access live videos on a daily basis, for investigation of daily incidents ocurred on the company's premises, the security boss depends on IT guys for technical assistance, etc. etc.)

(1)
U
Undisclosed #2
Aug 22, 2015
IPVMU Certified

How to prevent a Denial of service attack launched from the internal corporate LAN by rogue IT staff?

That could be a tall order! DOS is one of the hardest types of attacks to prevent, and that's when they come from the WAN side. Someone with access and know-how on the same broadcast domain could pound any open port until unresponsive, or probably take down the entire LAN.

But first, can you explain what the reasons are for thinking that DOS attacks are occurring? Is it just the video outages alone?

What do the logs of the NVR, PC's, switches say? A local DOS attack should leave plenty of evidence behind. Also, what's the motivation? Disgruntled IT guy?

You can put a firewall up and block every port except for 80, but no doubt they will be attacking on 80, so you'll need to restrict some other way. You could make a firewall rule based on ip, but they could spoof past it.

If I were you I would look for more compelling evidence of internal sabotage. Once you have that, then it may help identify the who and how. Turn on logging and realtime notification of network events.

If there is a staff IT person doing this, you really need to have them removed physically from the building ASAP, and have IT execute their sysadmin gone rogue contingency plan.

(1)
(3)
JH
John Honovich
Aug 22, 2015
IPVM

Excellent points.

"What do the logs of the NVR, PC's, switches say? A local DOS attack should leave plenty of evidence behind. Also, what's the motivation? Disgruntled IT guy?"

Agreed. Check this first to see if the logs back that up.

"If there is a staff IT person doing this, you really need to have them removed physically from the building ASAP, and have IT execute their sysadmin gone rogue contingency plan."

Agreed. This might be happening but seems a little crazy, so there may be a simpler, less sinister explanation.

Related:

"We've received many service calls from the end user complaining about the IP cameras losing video in the early morning hours between 12 midnight - 2 am."

Is it only during those hours? Is it every night or frequently?

UI
Undisclosed Integrator #1
Aug 22, 2015

the service calls are not every night, but every few days in between; and almost all after normal working hours.

UI
Undisclosed Integrator #1
Aug 22, 2015

Re: ...what the reasons are for thinking that DOS attacks are occurring? Is it just the video outages alone?

: Yes, the video outages alone. In essence, I think the VMS with it's webserver service running on port 80 (or any other port for that matter) is essentially "a website" whose only
role is to dispatch video to a browser for anyone to see live video (given, of course, that they have the user priviledge to pass the login screen).

Re: ...What do the logs of the NVR, PC's, switches say? A local DOS attack should leave plenty of evidence behind.

: I think we can discard the PC from the equation for the moment (meaning any user's PC on the corporate LAN that is) since any attack originating from any of these workstations may very well be hidden afterwards by the IT Manager, the IP gets changed from static to a DHCP pool of IPs to cover up, etcetera, etc. The rogue IT guy may have surely used a workstation that is not his regular one. Regarding the NVR; the VMS log that is running on the NVR only records username login/logout,date+time and if the login was successful or not. It does not capture attacks particularly. Where in Windows 7 OS do you say I can look for clues ?? I haven't been
able to find any evidence in the Windows Event Viewer (C:\Windows\System32\eventvwr.exe).

Re: ...Also, what's the motivation? Disgruntled IT guy?

: The integrator received an e-mail that basically said that the NVR was "no more than an assembled Windows PC with a couple more hard drive bays and fans running on it" and that it was unstable according to his point of view". Also, we know for sure that he favors Linux-based solutions for their "stability". If this IT is also a "proxy" for a competing vendor trying to sell it's stuff and "putting his feet inside the company" we don't know (and maybe will never know) for sure what deals are happening underneath.

Re: ...You can put a firewall up and block every port except for 80.

: The thing is, the IT Manager is the one tasked by the General Manager to be his sort of "technological advisor". From our point of view, he is judge and jury at the same time: meaning that he is judging the system to be unstable and unreliable and at the same time he is a normal user of the system accesing the recorded videos. When the system is down and holes in the recorded video appear he is the one making the service call to the integrator.

Our idea of adding a new hardware-based firewall is to isolate the NVR and present the system logs directly to the Board of Directors. In that case the IT Manager will have to explain why are there attacks originating in the side of LAN that he is directly in charge of supervising. The firewall would be a sort of "an impartial judge" so to speak. That's our main idea.

JH
John Honovich
Aug 22, 2015
IPVM

Interesting.

Have you checked the logs of your VMS / NVR? If there is such an attack, your logs should show a massive number of requests during that time. If you have, what do they show?

Avatar
Hans Kahler
Aug 22, 2015
Eagle Eye Networks

With knowing very little about this situation, I would look to see what else is on the network and see what other things are happening at that time. If it's for a defined time, it sounds like the type of thing that would be scheduled, such as a backup or software update. It could be that some management server is trying to update workstations and is getting confused along the way.

A DOS attack is possible, but given that it's in the middle of the night it seems like they're not really disrupting much, so what's the motivation? I would look to IT to see if they have any scheduled jobs around that time.

(1)
SM
Steve Mitchell
Aug 22, 2015

A denial of service attack is a real risk when attaching an NVR to the WAN (to facilitate remote access). It's one of the reasons I'm not in favor of that design/architecture.

That being said I agree with others here that it seems like a complicated conspiracy to conclude a DOS attack must be responsible for these outages without better evidence.

If there are no application logs from the NVR I'd look to Google for help with detecting a DOS attack in Windows 7.

You might also think about putting a network device between the system and the rest of the network not necessary to block/prevent the attacks but to log the activity.

UI
Undisclosed Integrator #1
Aug 22, 2015

If a hardware firewall cannot block/prevent a DOS attack, then is it safe to assume the corporate LAN is almost completely at the mercy of the attacker ?? ... I mean, if an IP or a complete range of IP addresses are blocked after an incident, the attacker basically only has to spoof another IP address or attack via a proxy VPN located on another country, is this correct ?? Which are the best practices in the industry against sustained and recurrent attacks spanning weeks and several months ??

(1)
SM
Steve Mitchell
Aug 22, 2015

A large scale, distributed denial of service attack happens across the internet and works by sending so many legitimate packets to the target that the networking equipment or host are saturated with traffic and cannot service any other requests. DDoS attacks use zombie bot-nets all over the world, and it's not possible to simply block an IP or IP range. So it's not that a firewall 'blocks' the attack, rather any mitigation would need to figure out how to filter the 'bad' traffic and allow only 'good' traffic. In the case of a professional attack, this is challenging because the network hardware needs to both handle a very large volume of traffic and know how to differentiate between the good and bad traffic. The current state of the art is to sign up with a DDoS mitigation service provider with a prearranged plan to transfer your service endpoint (via DNS) over to their network which has the capacity to absorb, apply filtering algoriths, and then forward legitimate traffic to your service for the duration of the attack. A lot of the companies that provide content distribution network services also have this capacity. A responsible service provider (i.e, any cloud service provider) needs to have a DDoS mitigation plan in place as part of their routine security assurances.

But it doesn't sound like you're talking about a DDoS attack from outside the internet onto your NVR but rather an internal attack, right? Thus there is no bot-net but rather you believe it's a rouge host or small cluster of hosts flooding the NVR's NIC with traffic and taking down its services that way, correct? If this is the case then it really should be possible to detect this traffic pattern and identify the culprit. The network just needs to be monitored to look for these attacks. And given that it should be easy to detect, thus the skepticism that this is the most likely cause of your outages..

Avatar
Billy Guthrie
Aug 22, 2015

Anything is possible! The likelihood of your video surveillance network experiencing a DoS is far-fetched. Like any other issues that occur on a network, you need to capture and analyze the packets to determine what is going on.

Some great points have been mentioned; however, not all DoS attacks are created equal. Some can be mitigated with the use of TCP SYN checking, Anti-IP Spoofing or too many connections from a single IP, while others such as DDoS attacks are very much harder to mitigate. If you are experiencing an attack, it is more than likely a DoS attack exhausting your layer 4 UDP resources; maybe even a UDP bandwidth volumetric attack causing 100% network saturation.

The problem with any attack on a local network, MAC and IP Spoofing can be used to hide the origin of the packets. Most video streams are UDP, It is highly unlikely that a few hundred botnets have been installed on the local network causing a Distributed Denial of Service attack; however, not impossible. As I said, you have to analyze what is happening on the network.

It is apparent that a video surveillance system was installed without any consideration of network security; always ensure that your video surveillance infrastructure (Dedicated or not) is secure. You need to segment the video surveillance traffic and any physical or logical connection made back to any other network needs to have some type of firewall.

We experienced something very similar maybe 5 years ago; around 1am every morning for 30-45 minutes, we lost video from all 32 cameras; not all cameras were coming back to the same IDF, six different IDFs were used. We were able to install Cacti, monitor all the switches and monitor all traffic from the port where the NVR was connected. Long story short; it was not some disgruntled employee causing some DoS (It did cross our minds); after using Cacti to monitor bandwidth graphs, we were able to observe a bandwidth spike (spiked to ~95Mbps) on the core switch (Where the NVR was connected to). Wireshark was utilized to capture the packets to determine the type and source of the traffic. It turned out to be a MS-SQL database backup. Every morning at 1am, a server on the other side of the building was used to dump a 30GB database file; on a 100Mbps network utilizing TCP and assuming typical overhead of TCP, that took around ~45 minutes. After hooking up a local USB drive to the server, the cameras never had any other issues.


Analyze the network and always secure your network!


Good Luck.

(1)
(6)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions