I have to start this thread.... My apologies if it's already out there. Recently we dealt with a large amount of tech support hours narrowing down what ended being an Axis issue. Nearly 20+ hours in labor paid out to target a vendor's problems. They were quick to point fingers at Milestone, but aren't partners supposed to hold hands and not finger point? On this same job, we found that Axis no longer has a backup and restore feature. Anyone else dealing with this one? What this means is should a camera fail, we've got to pay a tech to manually reprogram all that can't be done by a template- which is alot. I've never heard of a vendor cutting labor checks to snoop out their problems.... Anyone have any luck with this? Or are we all just one big beta test?
Are Large Companies As Good As It Gets?
And just so no one else has the same problem, it was an issue with the edge storage sd card format of ext4. As soon as this was enabled with or without the same stream profile, we lost anywhere from 7-10 fps on the VMS.
Ryan, thanks for elaborating on the comment. When you say you 'lost anywhere from 7-10fps on the VMS', do you mean the frame rate dropped (like from 20 to 10-13fps) or? I am assuming you hit this problem and it took a while to figure out what was causing this?
Yes, that's exactly what happened. On a casino, no less. We caught it before the system went live, but talk about liability...
Specifically happened on a P3354, with a Kingston Class 10, running OnSSI
Did they ever confirm what caused the frame rate drop? For instance, did somehow Milestone re-set the frame rate lower or did the Axis camera automatically drop the frame rate when sd card formatted with ext4 was used?
I will say, speaking as an organization that tests lots of new features, that crazy bugs happen a lot across all product lines for new features. Unfortunately, edge storage / support still is in that category. I am not excusing this but anytime we test something new, we figure that something weird / bad is going to happen. It does not always but it's not uncommon at all.
Well edge wasn't the primary storage, it was backup. So that at least was good. But it impacted the performance of the main stream, which is what the VMS records off of. Basically, as soon as the sd card was enabled, my fps dropped in system status of Ocularis (Milestone). This has nothing to do with the VMS other than that was how I was conducting the fps investigation.
What was the ultimate solution? Did new firmware allow using edge storage without any fps drop or did you turn off edge storage as a backup?
No, in fact on the P3346 units rolling the firmware back was what narrowed the problem down. ext4 wasn't available on 5.20. It is only available in the newest versions. Basically just using vfat fixed the problem, until Axis figures out why the fps drops. My guess is that the ext4 format takes more processing power to condense, and the performance drop is seen in the fps.
Ok, thanks. Now it sounds fairly clear to me. Milestone/OnSSI weren't part of the problem but someone at Axis along the way suggested it might be them.
Returning to your opening question, "I've never heard of a vendor cutting labor checks to snoop out their problems.... Anyone have any luck with this?"
We actually had just this discussion in the old LinkedIn group where I asked, "If a product is found to have a bug, should manufacturers compensate integrators for wasted labor?" It got over 100 comments and I swear a handful of manufacturer death threats :)
If you are an Axis partner, I'd ask them for credit for future purchases. Explain the amount of labor involved and how it involved an Axis bug, especially if this was not listed as a bug in the firmware release notes when you were deploying this.
Cutting checks is not standard practice but the better manufacturers, when they realize that a mistake is on their end, will do something to compensate integrators impacted.
Ryan, for backup and restore, do you mean camera settings? As in, I'll dump the camera's current settings to a file and load then back later if need be?
Yeah, as far as backup and restore goes, there isn't any with Axis. Backing up of camera settings has been gone for a little more than a year, and no eta for release. Axis literally gave us a website in the phillapines that an engineer that works for Axis created for a temporary solution. Straight from my IT Engineer, 'That's not enterprise. It's not even small business!'
Ryan, I asked Axis for feedback on this. They are quite responsive so I am sure they will review and have feedback.
I have too. I actually opened a case, and their level 1 tech support tried to close it out by saying, 'it's not available, we don't know when it will be'. I said, escalate it and keep the case open until the feature is released because my problem isn't solved until it is.
One other thing you might ask them is why they only test their cameras with their own VMS. A tech specifically told me on the phone today that Axis only tests their cameras with their own VMS. I'm sorry, but are you too big to test integrations? Milestone, Verint, OnSSI, DvTel all test their software with multiple camera manufacturers- why does it not work the other way around? Especially when you are the largest! I'm looking for the philapines website now...
It was actually Singapore, and the site is down now. So all those backups we've done in the past with this third-world site are useless I guess. Notice the date is July 2012, and still no backup options 2 quarter of 2013...
Any word from Axis? My guess is they will give you their 'template excuse'. You can create a template.... Not the same!!
Ryan, what configuration settings are you missing from the camera that's not templateable?
I find this interesting, as their cameras run Linux. Everything is a file in Linux, so everything would be capable of going through a backup routine. I wonder even if you can pull everything via SSH? I'll have to test...
Maybe a list of files to backup in the camera would suffice? I would love to store camera configuration files like Cisco switch configs...
There's a ton. From SNMP and alerts- such as email alerts, ASCII, etc (which can be very time consuming on larger projects), camera names, ptz titles and presets (on ptz cams and encoders), theres more than just this small list, it's just late. I had to spend nearly 7 hours last week manually reconfiguring a bunch of the above which couldn't be done via templates.
I've wondered about the whole lack of configuration backup before aswell and mailed Axis about it.
I had a Q1755 at the time and since they're quite critical in the installation they're in I was wondering if I could easily get the configuration out of it and install it on a new one in case one died.
They send me a simple Java program which could export the configuration and import it as well.
It hasn't been working perfectly though, but it something. Still find it weird though that it isnt just standard in the programming.
(Disclosure: I am a Bosch employee) We do support saving/backup of camera configurations in our IP cameras and encoders. This includes PTZ cameras settings (ie. presets, tours, etc...). The only parameters not included in backup config files are: IP address; time server address; and multicast.
Additionally, you may find our free Configuration Manager software to be a useful tool for these backup processes - as well as collecting and documenting necessary camera/encoders settings by exporting into a .csv or Excel friendly format. Especially helpful with large systems.
Ryan, Ethan sent you an email last week. Did you respond to him?
As for Bosch, their cameras are generally in a similar price range to Axis. They have the Advantage Line which is like the Axis M series and then their professional stuff which is like the P series (for both fixed and PTZ offerings). I am not saying you should switch, just that their products are not stupid expensive (e.g., from just last month, Bosch's new 16 channel encoder).
I just reforwarded you the email. Email john@ipvm.com and ethan@ipvm.com to discuss further.
Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.