Why These Bandwidth Spikes At Regular Intervals?

Can anybody help me out with this?

Attached is a print screen of an Orion report that my company’s network administrator sent. He noticed the spikes at regular intervals and is concerned about bandwidth. He seems to think it’s a configuration issue with the camera. I disagree and think it’s an issue with the camera talking to the VMS, or vice versa.

VMS is Lenel, OnGuard v7.0 and the cameras are Axis P3364’s.


What devices are on that link / network? Is it just the cameras and the recorder / VMS? Do you have any file servers, network based storage, etc. on that network? It would be useful to rule out any other devices causing this.

Also, how new is this? Any idea of when this started? If so, any idea of what, if anything, changed around that time?

To add a little to this... This is at an offsite away from any of our main facilities. It operates on a commercial connection (not sure if it is AT&T or Verizon) as opposed to direct through our own fiber. Looking at the upper right square titled the top 5 endpoints the number one dark blue is the VMS, number two light blue is one of the AXIS cams and number three is the other AXIS cam.

This is relatively new or we would have heard of it LONG before now. I will see if I can get them to run a history to try to pinpoint when it started.

Thanks for all the replies! I will continue to look at your responses here and see what we can try and answer questions as we can.

Spikes intervals are very regular and separated by about 20 minutes. Therefore, what we can say for sure is that they are not related with increase of activity in the image.

It seems that your camera, VMS or anything else in the same network is performing a scheduled activity, like a backup.

I suggest you to check that.

The pics don't really give enough info to diagnose this. Where is this being measured? Are the spikes showing data going to or coming from the VMS?

You could try running Wireshark on the VMS machine and check those logs to see whether this traffic is going to or coming from one camera specifically.

Going out on a limb, I'm going to say there may be a problem with the polling interval in or data aggregation with Orion/SMTP.

Loo at the top 5 endpoints graph how there is virtually no traffic for all of the hosts (except for the tan), and then all spiking to pinpoint peaks over minutes, at a rate of 25Mbps and 50Mbps for the cameras, and over 100 for the VMS (assuming that is the dark blue line). If you spread that traffic over the whole period though it seems reasonable.

Is edge storage enabled on the cameras? Could they be recording locally and then periodically dumping?

Going out on a limb, I'm going to say there may be a problem with the polling interval in or data aggregation with Orion/SMTP.

This would be my first line of thinking as well. It's almost impossible for a camera to push 150Mbps of throughput, especially when most only have 10/100 NIC's.

Programs that do these data charts work by getting a packet count from a switch port at Time A, and then another at Time B, doing some math and then plotting a throughput rate. If the sample time is very short (1 min or less) you get nice smooth graphs with decent resolution. If the sample time is very long (like 20mins), you'll see giant spikes...

Regular means not Jigle but scheduled I do agree

SNMP is consuming very low bandwitdh, won't raise that levels between 2 main stations (Server and main Client ?)

A FTP in Mpeg could generate also a raise but not that much from 0 to 150 mbits.

SD Thrickling.. (rare, because used at night) or PTZ camera (more common) hourly tours moving to new pre prepositions...mmmh ? (could x 5 regular bandwidth) or automatic hourly new videowall displays that will raise peaks till the system moves from High res Stream1 to Low res Stream 2 for mosaics purposes ?

I will say, both PTZ tours generate a general increase, to servers and then to display clients ...

Backup operations would last more time according to me.

Is there PTZ cameras with scheduled tours ?

is there WIreless transmissions ? (potential transmission buffer issues)

post production Analytics (if any) could also generate that , by taking streaming/ or FTP Moving some (not all) videofiles from video server to analytic server for content analysis. If the second station (14% bandwidth) is an analytic server it might be also a clue.

I 'm very interested by the final true answer if we can find it out ...

Weird. If you have access to the server logs on the VMS and/or cameras I'd look at those and see if there's any clues that coincide to the times involved.

Additional info on this issue. The first camera highlighted is the real problem, but included some others at the same site for comparison.

First column is obviously the date, second is total Bytes transmitted and received, third is total received and fourth column is total transmitted.

Ross, I noticed also on your first post that the spikes are on Ingress, and here you are showing all the cameras receiving far more data than they are transmitting. Is this a correct interpretation or are they actually reversed because of the sniffer's perspective or something?

I think your netmon software is lying to you. The camera has a 10/100 port (I just checked the specs) and from the switch port description it appears to be connected to a 10/100 switchport. There is no physical way you're getting 150Mbps of throughput on a 100Mbps port.

Something is configured wrong, and it's not the camera.

good point Mister B.

Ross, If your switch is L2+ with management, do a port miroring on the cameras ports, then on the VMS port, and measure with networx utility the real Up/dow throughputs from your PC.

(port miroring replicates selected ports to a listened port where you connect your PC and sniffing tools , so you can investigate without stopping the network)

"There is no physical way you're getting 150Mbps of throughput on a 100Mbps port"

Even in full duplex mode........?

Full duplex mode allows you to transmit and receive at the same time. So you could *theoretically* get 100Mbps in each direction, giving you 200Mbps of total throughput. The graph above is showing traffic in one direction only.

Actually just to clarify, the way that top endpoint graph was run shows the sum total of bytes that each endpoint sends AND receives.

The confusion may be over the term Egress Bytes. Since it's the switch that's collecting the data, everything is from the switch's POV. So Egress bytes for a given endpoint means the total bytes leaving the switch with a given MAC address, regardless of where it originated.

So when the camera is transmitting video, it's bytes get counted on Egress because they Egress from switch on the port that is connected to the VMS. And when the camera is receiving data from the VMS, these bytes also are counted as Egress for that endpoint, because they Egress on the port connected to the camera.

A simple test of this is that if the chart was only transmitting or receiving, then we only see the video traffic from the cameras or the VMS, not both as shown. Enabling a report to show Ingress and Egress bytes combined typically double counts the traffic. Here is the man page for it.

The last info provided by Ross however does breakout transmitted data and received.

Somebody is using a Mjpeg Stream from remote instead of a H264, or the camera is sending a large Mjpeg by FTP to a monitoring center or.. look at destination of packets with wireshark

see camera log to see if a script inside the camera launch a repetive task or if an external IP (VMS or somebody else) ask for a specific stream or read the SD or ...

@Ross, disregard my previous question about the ingress bytes.

@B, Although I agree in general that it looks like a reporting anomaly, I don't see 150Mbps on any individual camera, rather it would appear that it is the VMS that is going that high, with the light blue camera going to 75Mbps. Since the VMS probably is Gigb it doesn't prove conclusively that the reporting is wrong.

@Ross, the transmitter/receiver data that you included, while certainly higher for the first camera, is in plausible range, based on the Axis bandwidth calculator, which shows 15GB-50GB depending on the parameters, for one day with that model camera. We would have to know more about the differences in the cameras settings/scenes to know for sure if something wrong there.

Going back to the first top 5 endpoints chart with the spikes:

I think that the fact that there is approximately ZERO bytes shown for the cameras and the VMS most of the time proves that something is wrong. To a network admin who may be used to nodes which burst traffic sporadically, it needs to be explained that these cameras, even if they are spiking, must also be streaming, otherwise there would be no live picture. Assuming that the cameras are always viewable, the onus should be on IT to explain the discrepancy.

If these spikes are always happening, can you just bring up the axis web interface during one of the incidents and watch the Mbit/sec counter realtime?

@B, Although I agree in general that it looks like a reporting anomaly, I don't see 150Mbps on any individual camera, rather it would appear that it is the VMS that is going that high, with the light blue camera going to 75Mbps. Since the VMS probably is Gigb it doesn't prove conclusively that the reporting is wrong.

Even 75Mbps is a LOT of data for a network camera to sustain, that just doesn't pass the sniff test.

Similarly, if that top-right graph in your first post is the ingress to the VMS, you should probably see a sustained input of at least 1-2Mbps per camera. Not sure how many cameras you have, but I'm guessing enough to have at least 10-12Mbps of sustained data.

OnGuard 7.0 doesn't support edge storage, so unless you're running any other sort of service on the NVR that would support that (eg: FTP server) I can't imagine what would even allow a 75Mbps data burst from a camera.

I agree with you B, I just wanted to make sure we were both reading the numbers the same...

Sorry folks, was out on vacation and just digging out. I have gone back and forth with IT on this and as of Friday Tony pulled the cameras from the server. Just recording on SD cards until we can get this figured out. We made the top ten list of all traffic corporation wide. Which is not good because that puts us on a whole bunch of people’s radar. Friday of this week I am going to go to the site with one of the Network guys to put an eyeball on the whole setup. Hopefully we can get more answers by checking everything on site.

Thank you very much for all the comments, questions and ideas. It has helped us out greatly!

@Tony/Ross, any update on this one? :)

Unfortunately nothing really as far as answers. IT folks did not want to check all their equipment on site. So we swapped the camera with another of the same model and we now have the camera back being recorded on the server as of this week. We will see if the new camera does the same thing and then IT will finally believe us that it is not the camera causing the issue. Hopefully……

Me and the network guy went out and checked the actual switch this cameras is on and found nothing wrong with it. The bandwidth at night jumps up extremely high, which is not surprising considering its pitch black in this area. I asked the network dude about the fact that the camera is capped at 1.5Mbps. He indicated that the bandwidth will be much higher than what the cameras capped at because the network flow is "accumulative". He indicated that there is more traffic to and from the camera than just the video stream data.

Question for you network gurus: Does this sound legit? What's the point of capping a camera at a certain amount if the "accumulative" amount is going to be much higher?

Tony is too nice to mention that this particular conversation was a week after the IT group gave us a "lesson" in how IP cameras “actually” work, how they measure motion and why light, or lack thereof, does not impact performance of the camera if it is a “good” model. None of which was accurate.

Basically all we can do to hold our tongues. Last thing we want to do is upset the IT guys!

"why light, or lack thereof, does not impact performance of the camera if it is a “good” model."

Sigh :) You can share this with them: Testing Bandwidth vs Low Light, which includes stats from 'good' Axis, Bosch and Samsung models.

Tony is too nice to mention that this particular conversation was a week after the IT group gave us a "lesson"

Basically all we can do to hold our tongues. Last thing we want to do is upset the IT guys!

Print him this thread, tell him how everyone on it was 'fooled' also, and how we are all mighty appreciative of the 'lesson'. ;)

He indicated that there is more traffic to and from the camera than just the video stream data.

Easy rebutt: Ask him to define those additional traffic sources. Do the bandwidth calcs from there. You'll blow up the statement as the source of the issue pretty quickly.

No! Remember the point is to KEEP them happy! ;-)

What about their accumulative statement? Yeah, I am sure there is more going back and forth than JUST the actual image data, but that has to be included in the cap right? Even if it is doing NTP, working hard to process images at night and someone is logged in watching it live shouldn’t the cap be the cap?

Here is how we have it set, FYI

Our settings

Crap! I forgot to click the "accumulative" button to off.

Dang it Tony!!! We have been over how important that button is a thousand times!!!

The GOV number is dictated by the VMS.

XYZ.

It sounds like your Axis ZipStream has come unzipped. :)

Can you overlay bitrate on the video and check to make sure the cap is working? I've seen caps not work. It's rare, and I don't think I've seen it in Axis, but it's worth checking.

There is other data going back and forth, which generally is not included in the cap, to my knowledge. BUT, it's tiny. It's metadata, and sometimes control commands from the VMS. It should be a few Kb/s at most.

I agree. I wonder if he is getting confused that just because it's duplex comm capable doesn't mean it's always duplex at the same rate.

You have a cap on recording, but are multiple streams being used and are those settings optimally set?

Seems to be working at the capped level.

Maybe we should cut him some slack.

I remember my first job...

The cap isn't the cap of total bandwidth out. It's the cap on that stream. If you look at the Stream Profiles section on the Axis interface, it has normally four stream profiles by default:

Each of them has an individual cap setting. Maybe as a test try capping all of them low and see what happens. It doesn't sound like you're using multi-streaming, so not sure why they'd be in use, but it's worth a try.

And regardless of multi-streaming or whatever, nothing explains the enormous spikes with no activity in between them except for the ORION network 'accumulators' polling interval being set incorrectly.

OK I went back and asked for some additional reports/graphs. I asked for weekly, daily and hourly for the cameras. I am going to slap them on here as they come in.

Monthly

30 minutes

Weekly

Another week

Weekly of camera 5 to compare to camera 4 on other side of building.

Is it much less every weekend? And what happened the week of May 25-June1?

I'd start looking at what's "special" about all those nights that show lower bandwidth. Night shift vs. no night shift? Long weekend? Particular equipment that doesn't run on the weekend and was down for maintenance that one week?

Try looking at a monthly graph and look for repeating patterns.

I think we have tried so many things to get this under control we have lost track of what we tried when. Also weekends vary by Officer covering and they may or may not have had the camera(s) open. Sounds like it is time for a department email.....

What a difference in that graph vs. this one:

Now it seems more like a noise at night issue. Why the cap is being exceeced, assuming there is no multi-streaming or ftp'ing of files, is the question.

Now that we see that it is an every night type of thing I am thinking so too. However, I am not sure if it is due to night time settings or if it is due to the third shift Officer at the site having the camera up and running his whole shift.

Tony is going to get some more light in the area as a test, but looks like we have jumped through all these hoops and worked on this for months only to find that the cause is night time!

or if it is due to the third shift Officer at the site having the camera up and running his whole shift

You may be onto something here. First thing I'd do is find out if that person was away the week of May 25-June 1... if so, that's a strong correlation to the low bandwidth through that entire week.

We will give this a shot, but I am about 99% sure this was one of the first things we tried.