While Hikvision is heavily marketing deep learning and 'AI' as their next big thing, new IPVM test results of their DeepInMind NVR shows their deep learning works terribly.
We bought (via authorized Hikvision distributor ADI) and tested the iDS-9632NXI-I8/16S DeepInMind NVR [link no longer available] for a month. It suffered from serious issues across the board, frequently identifying, e.g., vehicles as women, rabbits as people, missing real intrusions, not analyzing certain demographics at all. Plus, it suffers from usability and tech support problems.
Despite this, Hikvision has taken to ADI's February 2018 flyer to tout its clearly broken DeepInMind offering:
Inside this report, we share video, gifs, and images showing the various problems plus examine how Hikvision can improve the system.
(1) If you are a 'thought leader' or analyst who got swept up in deep learning marketing, this should give you pause and some sense of reality. Doing this well in production, rather than a demo, is hard. Hikvision has a lot of work to do.
(2) The inability of Hikvision to properly QA this (it should never have been released in this state) and train their tech support (you cannot sell something your tech support is incapable of supporting) is a warning sign of Hikvision stretching itself too thin across too many products and market segments.
(3) Companies can obscure cybersecurity problems because they are hard to directly see. You cannot obscure video analytics one. It’s right in front of the user and if you categorize rabbits as people and trucks as women, people will quickly lose trust in the product and will not use it.
I ran this through my Hikvision DeepinMind NVR and it correctly identified him as not a rabbit. It also said he is an 10 year old Asian Female Ford Escape.
Sounds like it has potential but was released with too many features, way too fast without proper vetting. This was such a huge jump in features and is almost like a new product entirely, its going to take alot of training and R&D to get this right.
I'm confident with firmware upgrades that the product will work great. Nonetheless this was a botched release and Hikvision can do alot better. A similar thing happened with their doorbell. When we first received the doorbell, the firmware that was available at the time we received it was plagued with performance issues. Luckily the latest firmwares have solved most of the serious problems. But that was another case of a product that was released at a time when it shouldn't have.
I would advise Hikvision to use real world beta testers before releasing products to vet issues such as these. Their is only so much your in house R&D can do in controlled environments. Putting this out in real world scenarios with beta testers will give you alot more troubleshooting info. Also, I hate to even say this but please dont go the Dahua beta testing route which is "Release a product for sale and solve problems when reported"
With such a revolutionary product such as this, its important to make a huge first impact by getting as many things right as possible. Getting things right on such a new and exciting product will solidify your place in being a market leader.
When we first received the doorbell, the firmware that was available at the time we received it was plagued with performance issues. Luckily the latest firmwares have solved most of the serious problems.
While I agree the analytics will get better, making analytics work well is a lot harder than a doorbell. There's hard R&D problems with analytics whereas a doorbell is straightforward feature development.
Of course, we have to wait and see.
Sean, I am curious. Let's take the optimistic side and assume it works really well by the end of the year. Where do you see such a product selling in your customer base? Is there much demand? If so, any specific segments?
I agree about the differences in complexity about doorbells and analytics. I was just stating that the method of "release a product and deal with problems as they come" is not something I particularly care for. This is very Dahua "esque". Hikvision is better than this.
We target the small biz and resi market so this product would be a particular slow seller for us. However we do have the occasional commercial job and this would be a great product to advertise and provide. The advertising alone of the features would have been a great tool to bring in new customers, even if they didnt buy the particular NVR.
The downfall of this is a missed opportunity for Hikvision. As you know Hik wants to make more of a move in the enterprise market. This would have been a great opportunity. Had your review been glowing, this could have really caught the interest of some integrators that service this market segment. I do feel they will get most problems solved and it will be a fairly solid product but I hope its not too late by the time they do. I beleive the key for enterprise customers is super solid software features. Hardware is great, but software software software is the key.
I largely agree with everyone on this. Its something we're looking at, but to be honest - the customers aren't screaming for it. Retail has a good use for it, whether its searching for lost children in a shopping centre or a shoplifter. The problem is that the money is not there anymore as the high streets are increasingly squeezed on profit protection.
One issue we did have with Hik analytics is that settings available within the camera are not available within IVMS or NVR GUI. Specifically the ability to define a max and min target size for intrusion, line crossing etc. When these are set within the camera the accuracy improves significantly. I'm told this is being looked at in the IVMS, but don't think its available yet. It perhaps may have been an issue in this test in regards to the targets missed on line crossing/intrusion?
Other tweaks with analytics would help - intrusion only triggers recording for a pre-predetermined time period rather than for the duration of the intrusion - annoying.
False alarms from rain drops is any issue for any system this isn't going away fast. On our larger systems we kill the camera IR and flood the scene with separate LED IR lamps - reflected IR from rain drops then disappears.
I can be positive and say that at least Hik have come to market with something that certainly has the potential, so they can hardly be accused of copying. However, it does seem a case of bleeding edge technology and not cutting edge and a marketing fail which is a shame because the effort and ambition is there, but the delivery is bad.
It should have been pushed out to integrator to beta test first. In fairness, I don't know of other CCTV manufacturers that do beta trials - but this is an example of where this may have been a good idea.
I do feel that as Dave D says, improved firmware is available and within the context if PIR triggered systems, the AI is a major step forward that shouldn't be overlooked. Even using the less than perfect intrusion and line crossing, we have reduced false alarms massively.
On an unrelated Doorbell sidenote... I found it humorous when HikVision USA purchased an Axis A8105-E from us back in mid 2017. Its no secret that manufacturers buy and test their competitors, but we got a good chuckle out of this at the time when no Hik Doorbell was on the market.
Nice, thank you. Yes, I knew there were plenty of Doorbell/Doorstation prior to Axis, with Mobotix coming to mind. We were just surprised to see the Ship To address as Hik.
But they won't even let me have the DiM NVR to test because it's supposedly no where near ready.
#9, good feedback! Any idea or theory about why the variance across countries? Clearly, Hikvision is selling it in different parts of the world beyond China and it's being promoted for sale and shipped in the US.
In the past, I've gotten the impression that there is some local autonomy about what, and equally importantly, when products are released in a country. And that makes sense to me. What does not make sense is why Hikvision would be shipping / selling this anywhere? Did this legitimately pass Hikvision HQ QA?
P.S. - this was Ethan's test, I just assisted in contacting Hikvision and checking the sales related issues.
Outdoor analytics are very challenging, that it didn't farewell is no surprise.
The indoor tests, where the analytic may actually prove beneficial, lacked some detail that would allow readers to judge its practicality. PPF is not mentioned. You gave an idea of the false alarms for exterior use, but not anything for interior.
An analytic that could detect people and a few other items for interior use may actually work and be beneficial, well at least for me that is.
That Hikvision bit off way more than it could chew is obvious. I am also suspect that the upgrade will make everything work. It might correct some of the issues, but I seriously doubt it will have it at 100%.
There is no AI or machine learning without analytics that work, are usable and produce meaningful output. I don't understand why the industry doesn't tackle this first and then move on to AI.
At the point where subjects enter/exit the hallway in the near field (highlighted), PPF is ~110.
Where the side hall enters about halfway back, PPF is about 60. So at both points, pretty strong. Angle of incidence was also low for better detail shots and proportions of humans.
Results were better inside, but still had issues. For example, this shelf of file folders triggered multiple alarms:
On another inside camera, an empty chair, door, and doorframe all trigger alarms as a door beneath the camera is opened.
Obviously the person opening the door is a valid event, and he was captured in a separate detection prior to this, but these objects should not be triggering when false alarm filtering is turned on.
We have one of these installed and experienced similar results. I am blown away that HIK is advertising and selling this now. It should be considered in beta test mode until they improve the analytics.
We experienced multiple false alarms daily, always from rain and vehicle headlight reflections. The alarms got progressively worse over time leading us to believe the DVR was learning from its own bad behavior. We now affectionately refer to it as our rain detection DVR.
No amount of tweaking configuration fixed the false alarms. We love the potential this product offers but HIK needs to get their act together quickly or risk getting a bad reputation for releasing unfinished products. (HIK-connect anyone?)
Its amazing to me that anyone is claiming "This product has potential". Thats like a jewler selling coal but marketing it as a diamond and a customer buys it and says, "It has potential, I just need to wait"
AI and deep learning are being worked on by many camera manufactuers and software developers. What a shame a manufacturer knowingly releases this lump of coal to trick people into thinking they are innovating and will now increase buyer scepticism and delay the adoption of new technology (when it is ready) that may in fact save lives.
Note: Several members have contacted us with offers of newer firmware. However, we are hesitant to test these versions without being 100% sure they are intended for our North American model, so we plan to wait to upgrade until official Hikvision NA firmware is released.
We plan to retest when firmware updates are released.
This is not surprising. You can go to Hik's different websites and see products that they've come out with in other markets. When you ask regional techs or sales reps about them, they don't have a clue.
I called Hik once to ask when this newfangled 4K Darkfighter PTZ that was on their European page was coming to North America. They had no idea what I was talking about and couldn't even find a data sheet on it.
But this NVR is shipping in North America, from multiple distributors on their authorized list. It's not a matter of when it's coming to North America. It's here and needs to be supported.
Them telling us that they won't support it for an "end user" would've been more respectable than simply being so unprepared that they can't.
I'm not defending Hik so I don't quite understand your response. I'm criticizing them for not even knowing their own product line, never mind supporting their hardware.
Great example of how those "deep learning AI" buzz words features sounds great on paper or in a well prepared demo scenario but perform below acceptable standard in reality.
I just spoke with someone from Hikvision USA about this and they informed me that this was not supposed to be scheduled for Final Release in North America. They said that it is scheduled to be released in March but probably wont even be released then due to more testing needed. They said that a few units have been shipped to "certain" ADI branches for "field testing" for certain channel partners and sounded unsure of how you even obtained the unit.
If you go to Hikvision USA's website, this isnt being shown on their website either: http://www.hikvision.com/us/index.html
How did you obtain the unit by chance? Was there anything said to your purchaser that this was still in testing or anything like that?
It's also being sold in Europe, in Asia and other parts of the world for many months.
We don't have any special connections to Hikvision or access to 'testing' or beta units. We simply ordered through one of ADI's thousands of dealers. There was no disclaimers nor limitations nor warnings given.
Not only that, I contacted Hikvision literally a month ago letting them know we were starting testing on the unit. In the week before publication, I left at least 6 messages with various Hikvision senior management people, letting them know that the unit performed poorly and that tech support had been unable to help, encouraging them to talk to us and give input.
Sean, your Hikvision contact is doing damage control. They know it does not work and since they cannot argue our test results are wrong, they are taking the incredible excuse that it's our fault that we bought it through their authorized distributor.
Thank you for helping to expose the hypocrisy and spin that Hikvision tries. Please take this back to your Hikvision contact and see what other types of excuses they try.
This could potentially fall under False Advertising and get them trouble, if people start challenging them about it and they can't provide supporting proof for their claims.
Yes, I think it does. The FTC, though, only gets involved with a fraction of such issues.
But Hikvision is building a track record of questionable / overstated claims.
For example, Hikvision USA January 5 post: "new Hikvision DeepinMind NVR with the human detection feature. The iDS-9632NXI-I8/16S nearly eliminates false alarms from non-human subjects"
Seriously, their Marketing executives should know better. I'm taking names of who to not work with or for... it comes down to an ethics problem, not just a legal one.
With the absolutely filthy online sales channels Hik has in North America, how could anyone be surprised IPVM got their hands on this? Amazon is literally flooded with Chinese sellers of Hik products.
Yes, the entire article was read yesterday. I especially liked the part about the leather stitching in the pants seam.
We have yet to replace an Axis camera that turned to Mandarin after a firmware update. I would also venture to say that Axis North America will offer support to any product with factory warranty left. Whereas Hikvision North America literally said "we cant warranty that" because the camera originated in China with a Chinese Serial. This customer has several Hikvision branded paperweights we had to replace.
By definition "deep learning" implies that the algorithms would adapt and learn the differences between false positives or identifications and therefore dynamically improve it's results.
That's a very misleading label to slap on an NVR with a static group of analytics run by an outdated GPU
By definition "deep learning" implies that the algorithms would adapt and learn the differences between false positives or identifications and therefore dynamically improve it's results.
I understand where your coming from, but there is a important distinction between when and where the neural network is created.
In this case at least, it would appear that deep learning was only used in the creation of the neural network, i.e. back at the lab. The static neural network is accessed by the device in real time at the device.
Since this method requires much less computational power we should expect to see it in many of the early so-called “deep learning” toasters.
Hopefully this half-step implementation will soon be unnecessary.
Im getting a little fed up about the half assed solutions they spew out.
Out of all the "DeepIn..." products I have tested only one seems to work properly, the people counting camera iDS-2CD6810F/C. The face recognition camera iDS-2CD8426G0/F-I is rubbish and so are the NVRs.
I am curious about a few things. Typically "AI" or analytics takes time to configure or learn in. I don't have a lot of experience but I have done a Panoptes system and the software required a lot of time reviewing pictures to give examples as to what a human was and what a car was. Is there anything like this on this Hik machine?
There is nothing in any documentation we have seen (both public and otherwise) that indicates learning time is required. However, before we started putting together this report, the NVR had been running for a couple weeks, and has continued to run since, and is not improving.
A week after the examples in this report, I just scanned through video from the weekend and the NVR alarmed on snow, foliage, light changes, and various other sources of false alerts.
Would you say at this point that if there was a shootout between a camera tuned up for pixel motion detection (proper thresholds and filtering) it would be of similar, worse, or better accuracy?
I would way it's still better than pixel motion detection for rejecting non-human sources. Even properly configured VMD would still trigger on rabbits, SUVs, foliage, etc.
However, before we started putting together this report, the NVR had been running for a couple weeks, and has continued to run since, and is not improving.
Nor is there (apparently) any mechanism to give the machine feedback on its choices, which also indicates it’s a fixed NN.
John will probably recognize me as a Hikvision shill (kidding, though I do like their products), although I have to say this result does not surprise me. Hikvision does several things right. Good-performing, affordable cameras is one of them. Analytics is not.
I had a 4MP varifocal Hik dome under my soffit because I was having trouble with trespassers. I set up intrusion detection in a very specific zone and email alerts so I would get notified if someone was walking up the driveway at 3AM. This is admittedly a very low level form of analytic but the cam would constantly send false alerts from shadows, headlight reflections, or other nonsense (literally I would get an email with photos of nothing there). When it DID finally catch an event I wouldn't get the email for several minutes. Better yet, I set up a schedule for the cam to send alerts from roughly midnight to 7 AM and in the middle of the day it would be ignoring the schedule entirely and sending alerts.
I turned it off and resorted to other measures besides the camera.
I don't know, could be just about anything. Maybe my phone just wasn't getting the push notification promptly. In any case the camera failed miserably at doing a very simple analytic, behaving more like simple motion detection and COMPLETELY ignoring the schedule I set for it. I was not impressed for what the unit cost.
I was going directly through Gmail (i.e. logging into Gmail's servers with the exact same email I was sending it to; me-to-me) so not likely. I think the camera is just slow.
Hikvision does several things right. Good-performing, affordable cameras is one of them. Analytics is not.
That's a reasonable conclusion. On the other hand, from a global / stock / financial perspective, Hikvision is betting hard on AI / video analytics. That's a very big part of their story to investors and the public, i.e., they are not some commodity camera company, they are an AI company that happens to also make cameras.
Hikvision USA Inc., a leading provider of artificial intelligence, machine learning, robotics and other emerging technologies, and the world's largest manufacturer of video surveillance products and solutions
Notice, AI is first, and video surveillance is last.
They have a lot of issues to fix, corporately and otherwise, before they should go making that bold of a claim. Clearly analytics and "AI" have a LONG way to go before being a viable mass market offering.
It’s great to see how many of you are interested in ML and Ai. We have been working on this for some time now, figuring out a way to bring this technology to market in a scalable, affordable way that actually works. We have partnered with some of the brightest minds in this field and the results are nothing short of amazing. I see people asking is there a need/demand, will customers want this? The answer in my opinion is yes, they will and do want this. We have many types of customers that are currently using this type of technology with very good results.
There are many interpretations of Ai/DL. Below is how we define our offering.
ML is an umbrella term that includes a variety of techniques that have driven radical advances in Artificial Intelligence (AI). Some kinds of ML - like Deep Learning with Convolutional Neural Networks (CNNs) - get the most attention because their results are amazing and they require such massive computing power that the tech industry sees a sea-change opportunity in enabling computers to understand what’s happening in the real world.
The big innovation here is in applying ML to all video streams at a scale and price that has never been possible before. Our multi-stage processing funnel eliminates spurious motion, adapts to the scene, ranks the importance of events, learns quickly from very few user interactions, and applies labels in real-time at low cost. The processing pipeline is not only extensible with hooks but also employs a variety of ML techniques including Recurrent Neural Networks (RNNs), Long Short-Term Memory Recurrent Neural Networks (LSTMs), and the most expensive CNNs. This enables people to exploit the latest advances from Google, Microsoft, Amazon, NVIDIA, etc… that would otherwise be extremely expensive. People get a real-time Search Engine for their own private video streams. It's a browse and search experience that is unprecedented in both speed and cost.
If anyone would like more information please message me.
Our multi-stage processing funnel eliminates spurious motion, adapts to the scene, ranks the importance of events, learns quickly from very few user interactions, and applies labels in real-time at low cost...
So to be clear, this unit/firmware under test here is designed to train the NN locally?
Whats the feedback loop mechanism to do that, i.e. “NVR, that is a fish, not a man”?
The current feedback loop focuses on training NNs on the *importance* of events. Those NNs are different than the ones that do the labeling of objects, but they're key to cost efficiency. They learn by observing what you play, share, save, or give thumbs up/down.
The "fish vs. man" feedback loop is via the confirmation and removal of labels that have been applied to specific events (you click a label to confirm it and remove a label to say it's incorrect).
The current feedback loop focuses on training NNs on the *importance* of events. Those NNs are different than the ones that do the labeling of objects, but they're key to cost efficiency. They learn by observing what you play, share, save, or give thumbs up/down.
What do you mean?
We’re talking about an NVR, right? Is there a way to thumbs up/down an event on this recorder?
Ethan, just a couple of quick questions. Do you have any comparative results of the various cameras you used? The datasheet recommends the 4-line cameras and I can see you used some - so wondered how the performance compared against non 4(6)-line units such as the Axis, Avigilon and 2-line?.
On you intruder analytic screen shot, it doesn't appear max and min object size was set up. As I understand it, this is usually managed within the camera, so I'm unsure what effect bringing it into the NVR has (if any) - but when we've optimized these settings in the cameras before - the reduction in FA's was instant and extremely effective (with the issue i mentioned previously that intrusion is simply a timed trigger rather than a recorded event for the duration of the cause).
Performance across various camera has been pretty consistent. What triggers one is highly likely to trigger another. We actually had more cameras outside than what is shown here, but didn't test with them for as long, and performance was pretty similar. That included 2-line Hikvision, 4-line Hikvision, Axis, and Hanwha.
We did not set up max and minimum object size, for two reasons:
First, someone shared a document with us with some setup guidelines from Hikvision (which we cannot share publicly at the moment) and they include minimum and maximum pixel sizes, and we were well within that range.
Second, they don't mention using the max/min settings at all, so we omitted them. It also would seem that deep learning should not require these settings, as the NVR should be intelligent enough to look for proper size and proportions before flagging something as human.
Just an idea, I would love to see a test showing comparison of Hikvision vs. other Deep Learning/AI platforms. I know Avigilon has been touting their solution as the best thing ever as well. I'm curious to see if anyone has done this well. Working primarily with retailers, this is something they've asked us to deliver. My current answer to them is "it's not ready yet". Articles like this show me that overall sales and marketing campaigns are still more important to some manufacturers than producing a quality product.
Was the Hikvision unit removed from the box and immediately tested? If so what were the expectations of what it had previously 'learned'? It is my understanding that this machine will 'learn' (hence deep learning) but really knows zero out of the box. Thats why its called 'deep learning' .
We have a section on that in the report (below), and the NVR continues to run and continues to show false alerts daily.
Note: No Learning Period Mentioned By Hikvision
Note that Hikvision does not mention a period required by the NVR to "learn" each camera's scene and differentiate between humans and false alert sources. This is not found in any manual we have seen, nor was it mentioned by Hikvision tech support or multiple dealers we have spoken to over the course of testing.
That being said, testing was conducted over a period of weeks, and the NVR continues to analyze the same cameras shown in this report. If performance should notably improve, we will update accordingly.
Im am not trying to be a smart ass being fair and reasonable to Hikvision it would seem that this unit would have to 'learn' before it is assumed it is 'terrible' at recognizing things such as humans or cars or rabbits. Just seems like a reasonable conclusion as it is a 'learning machine'. Of course it is understood that there was a 'minor slip-up' and someone perhaps misplaced the page regarding "must learn first" before testing. There is an issue with the Chinglish translation where some things are misinterpreted as well, so lets dont blame Hikvision too much for a minor 'oversight'. Hell, Ford never put out a warning to sit on the Drivers side when operating a car, everyone 'assumes' you should, right? Lighten up Ethan.....
I'd love to agree with you, but the DeepInMind/DeepInView/DeepLearning line should work out of the box. As more testing data becomes available the unit could get better by applying a FW upgrade. No selflearning algorithm though..
So what do you assume is 'pre-learned" for you in your part of the world? Why would it be referred to a 'deep learning' if it already knows everything it needs to?
Im just curious so please dont take that as an insult.
We assumed the same as you did upon first introduction. A device with capabilities to detect and to improve itself. Only after more talks and more asking did we come to the knowledge that it's smart, but only gets smarter with FW.
#14, whether it knows nothing or learns over time, both options were covered in our test. If the unit would have improved over time, we would have simply reported that the first x days or weeks were bad but afterwards the unit improved. It's been over a month and running with no material improvements so that's not been the case.
As I said much earlier in the thread, we reached out to Hikvision repeatedly before publishing. We also reached out to key Hikvision partners as well (not Dave Davies feedback in the original report). Hikvision needs to step up and take responsibility. If there are ways to improve the unit's performance, let us know or they can let you know and you can tell us :)
If this product requires a learning period to perform as designed, and this isn't mentioned in any manual, install guide, or support documentation, this is a major screw up, and not a minor oversight. Common sense would imply a "Deep Learning" device would need to "learn" but only a few of us lucky ones are blessed with common sense. There absolutely has to be some kind of guidance or documentation on how this device learns, what time frame it needs to learn to be effective, and what the installer/users can do to help it learn. By assuming it just needs time to learn with no other guidance, how do you plan your install, or set a timeline with your customer? "Sorry Mr/Mrs. Customer XYZ. I know it been a month, but your system is still learning... I think."
This particular unit has had plenty of time to burn in, and nothing has been learned. With no instructions, warnings, or disclaimers, we are only left to assume it will work as advertised out of the box. This unit is neither learning, nor providing any assistance on how Ethan can help it learn. I dont know what else to call it but a failure, or at least a premature launch.
I wouldn’t disagree with this at all, but I think we are also missing additional culpability.
ADI are knowingly distributing a product that they can clearly provide no technical support for. This to me is a bigger story as they are US based and have the ability to make their own mind up NOT to sell something they feel isn’t right. Indeed, ADI are the teal issue here. If they had said no to HIK then that would have been a major impetus for HIK to sort the issues out.
If Dave D in the U.K. is the best tech support Ethan could find and the U.K. has access to better firmware for the AI then something is seriously screwed up between box shifter ADI and HIK in the US. It probably also shows that the U.K. distributors possibly challenge HIK more than ADI. In the U.K. ADI is just another distributor and not a very good one at that. Their tech support for HIK is very poor so that part of the business model seems to have transcended the Atlantic.
im not defending HIK on this one, but as we appear to have a better product available in the U.K. and using DVS tech support, we don’t appear to be in the same place as US integrators and Ethan. And that is simply down to ADI.
ADI are knowingly distributing a product that they can clearly provide no technical support for.
From what I understand there is more to the story. I admit, I dont know enough to have a super solid argument but I do know there is more to the story. I was told that this has not been fully released yet in the USA. The units that ADI have are for field testing for key partners. Last time I had access to ADI's website, that particular unit said "Special Order" and even said "discontinued" at one time. If that's the case, this article wouldn't hold alot of weight.
As far as upper level contacts at Hikvision not responding to IPVM. I think its pretty obvious and goes without saying why Hikvision would never respond to IPVM. I don't think its any surprise to anyone. Honestly, I'm not sure why IPVM wastes their time anymore with this.
Nonetheless, if this was released as a field tester, it does seem that Hikvision could have had better control over the release of this and I think they know that too. More than likely some loose communication between Hikvision and ADI. With that being said, I will go ahead and beat IPVM to the punch. "Sean, do you have any evidence to support your claim?" "Sean, you are just taking up for the fact your partner released a product that didn't (blah blah blah)"
I was told that this has not been fully released yet in the USA.
Well, they have certainly been heavily marketed and hyped in the USA.
The units that ADI have are for field testing for key partners.
Even if that is true, it clearly is not even ready for field testing yet.
No matter how you slice it, Hikvision has overhyped DeepInMind, and if you are right sent it out for "field testing" long before it was truly ready, which means they expected their customers to PAY for alpha units and then dedicate more free time and resources to testing and catching issues that Hikvision should have taken care of on their own.
And why does Hikvision need to go through ADI to get units to "key partners"? Surely they would want that tightly controlled in terms of which "key partners" get units, no need for ADI to be a middleman here, unless Hikvision needed ADI to bill these partners for the privilege of getting an incomplete product.
Honestly, I'm not sure why IPVM wastes their time anymore with this.
(1) We try hard to let companies provide us feedback so we can reflect and share that with our members. (2) Their continued non-response reinforces Hikvision's self-inflicted damage.
if this was released as a field tester
Not a field tester, just a regular production bought product.
Honestly, if I was CEO of a large manufacturing company, I would instruct all my employees to never speak to IPVM. Luckily, im just an owner of a small business so I can blurt out all kinds of stuff as loud as I want to. (there is an ally oop for you, feel free to slam dunk with statement such as "you got that right")
As far as Hikvisions example, their is absolutely 0 benefit for them to speak to IPVM. You have annihilated & vaporized that bridge along time ago. I honestly doubt they will respond to you anytime in the near future, if ever again. I also doubt they care about your opinion about whether its beneficial for them to do so or not. Given the fact that this site has a strong reputation for being a non-objective Hik Hater site, I think they could care less about winning IPVM over. I'm not sure why you even report that they don't respond to your inquiries, because everyone be like "Duh". The sky is also blue, but no need to report on that.
It's a measure of DVS that Dave Davies offered information and support to IPVM despite the torrent of IPVM negativity against the HIK brand.
John makes no reference to this other than in passing, which is disappointing - but not surprising.
It's convenient for IPVM (certainly their page reads and subscriptions) to have a whipping boy which is unlikely to change. John will say they give IPVM all the reasons to attack them and their is some truth in that. But the balance is out of kilter and has been for a very long time.
John says they report objectively on Hik - which we all know is not true. He still fails to apologize for the appallingly bad, unprofessional, objective reporting of the Hik camera in the toilet which really underlines IPVMs true agenda. Absolutely any story that can include the work Hikvision in any way negatively is food for the IPVM gods.
If IPVM were to report on the number on successful prosecutions based on HIK CCTV footage that would not have been available at a higher installation price point - the myth that is being crafted would vaporize.
Continuing this speculation about "field testing" units is silly. It is demonstrably false.
ADI is not the only distributor selling it. I'm not going to go down the list of every single one on their authorized page, but there are at least three with active listings of the DeepinMind NVR that I found in a matter of minutes.
So did Hikvision just have "loose communication" with all their distributors about DeepinMind availability?
Second, even if that is what happened, all Hikvision had to do was tell us this was the case. This is the most basic reason manufacturers should communicate with us. It is not risky nor is it time consuming to write one sentence in email to tell us so.
Additionally, "Special Order" doesn't mean it's not available. It means it's non-stock. There are literally hundreds of Hikvision SKUs that are special order at ADI.
So just remind me again Ethan, why wasn't your supplier - ADI approached for either tech help or advice regarding the issues you have experienced. Or even a comment as to why, as you suggest, Hikvision have put out a unit that has not been tested properly and as John has said has been marketed using deception. With this thoughts in mind I'm astonished no-one has picked up the phone to ADI.
Virtually all of your article is very good but is being undermined by the fact that you never sought help, assistance or comment from ADI that may have gone someway to either helping you or even justifying Johns increasingly hostile claims against Hikvision.
Seems odd that John would seek help from those he has alienated and yet overlook the offer of tech support that ADI splash all over their marketing. When I say "odd" I am of course being facetious as the reasoning is obvious.
Have you ever contacted ADI for tech support? They 99% of the time especially on a new system like this would forward you to the manufacturer. They don't have the capability or the knowledge to support systems like this.
Tim, if you want to feel that our testing is undermined because we didn't call ADI, that is fine. But it's perhaps the most asinine position I've heard.
We contacted Hikvision's own tech support. They were perfectly pleasant to us but had no answers.
I personally contacted tech support managers at Hikvision.
We reached out to product management.
We spoke to integrators who are testing the NVR as well who are having the same support issues and seeing the same performance as our tests.
If, in the face of all of that, you feel our test is undermined by not calling ADI's Pre-Sales Support Systems Group (the actual name), that's fine.
Well, yes. I feel that the article is undermined by not even calling anyone at all at ADI. They may or may not have had an answer - but you would would never know that without calling them. They may have an internal escalation procedure that may have helped, against we'll never know.
Why would it be referred to a 'deep learning' if it already knows everything it needs to?
Deep learning does NOT mean the system is required to go through a learning phase post-installation. It is a method for training a neural network to accurately classify input data based on previous analysis of related and/or unrelated data.
For video analytics in security/surveillance, Deep Learning is typically used to describe a system that can detect people and vehicles in various forms/orientations/positions. Sometimes this is a singular class (all people are classified as "Person" regardless of race, gender, age, size, etc.), and sometimes a Deep Learning system can classify a Person into subcategories (Male, Female, Young, Old, etc.). These systems are classified as Deep Learning because the manufacturer has trained the neural network in an R&D setting by (hopefully) feeding it 10's or 100's of thousands of images taken from surveillance cameras so that the neural network would learn from representative data.
Now, it is still common for deep learning analytics systems to require some amount of on-site/per-camera configuration. In many cases this is done to help the system sort objects by rough size or position in the field of view so that it can better ignore uninteresting objects and spend the majority of its CPU/GPU time on only analyzing moving blobs that have a higher probability of being something interesting.
In a sense, what Hikvision appears to be attempting here is using the camera for a coarse classification, and then use the NVIDIA GPU in the NVR do a deeper analysis on only the "interesting" stuff. It is an interesting approach to reduce the cost of the cameras by centralizing the computing power in the NVR, similar to how some analytics companies in the past have used an agent on/in the camera to do the coarse analysis and then used server resources to make final classifications and do rules processing.
Don't worry, I'm not mad, just laying out facts. Totally light over here.
You asked if it should be allowed to learn first. I told you that Hikvision didn't indicate in any communications with them (or their integrators and distributors we spoke to) that it needed to. Their documentation doesn't say that it needs to (both publicly available docs and others which have been shared with us).
But that being said, the NVR has been running for weeks. It was running for weeks before we started taking examples for this report. It continues to run. And I continue to check performance. We will update with more info if performance changes.
In other "deep learning" analytics that I've used in the past you set bounding boxes to show height of a human at 3 different distances, close, mid and far. The analytic would then calculate what a human for all distances in the FoV. Do you think that type of configuration would have made the learning piece of this a lot more successful?
On a side note, I'm curious how the algorithm can guess that a car is male or female. Using other Gender/Age/Demographic analytics, they've made mistakes and even had an "unsure/unknown" result but never tried to classify inanimate objects in my tests.
I'm curious how the algorithm can guess that a car is male or female.
The underlying problem there is that DeepInMin 'thinks' those cars are people. But, good question, I don't think we can even guess at why DeepInMind is determining genders of cars.
In our experience this does make a significant difference. I do accept Ethans point though, that nothing in the Hik set-up literature or seemingly woeful US support would suggest it.
I haven't used the deep learning, but my assumption would be that in learning there needs to be education, but its not clear how you educate the kit at all. Setting target sizes max/min made a dramatic difference to our analytics on HIK systems and was totally successful on the intruder and line crossing - but you do need to play with the two thresholds as well to get it right. Its not unreasonable to expect that the AI should be doing this already - but I'm guessing it needs prompting. Who knows?
"Deep learning" how do you define that? Already learned, NO....
Without any instructions its clear that in some way shape or form 'something needs to be learned'.Rest assured my opinion has not changed, nor will it ever, and I believe that you are fair in your report I am not dogging you personally but something is possibly missing here, could be the Chinglish thing striking again? No matter Hikvision will work out whatever kinks 'they determine' are necessary, if any, and this will evolve into a first class product along with all the others.....
I work very closely with a manufacturer who is developing deep learning analytics. While they are definitely in an "Alpha" stage right now, the test results have been exceptionally encouraging. Where traditional algorithms were giving a successful alarm rate of 90% or so, with about a 5% "alerted, but wrong" rate and 5% completely missed incident making up the 10%, the deep learning product is seeing a successful alarm rate of over 99%, with only 0.1% missed incidents.
Testing and tweaking continues. This is the way a true technology leader works. HIK, on the other hand, releases half baked products.
Achieved, no doubt, in some sterile friendly environment and not the hostile complex outdoor real environments that integrator's work in.
Easy to say Hik is half baked - but on the assumption this "solution" doesn't even exist at hardware level - you are in no position to compare with your "on paper" solution with Hiks, badly released actual version. If you had pride and confidence in your "Technology Leader" claim - you'd be happy to share the manufacturer name, surely?
This test is taking place in a live, real world environment, working with an existing customer. Existing analytics were replaced with the more intelligent solution - same camera, new software. Suffice to say, this is also a purpose built camera.
I'm going to go out on a limb here and assume you didn't read my entire post, or didn't consider the post very closely. There are myriad legal gags and restrictions when two manufacturers are co-developing a new product, especially a product that is still in early development. Responsible manufacturers complete the development process before releasing or revealing. HIK, it would appear, has taken a different route.
So part of being a responsible manufacture is to state your competition is half baked whilst not having a shred of evidence to support your self inflated “product”. Hiding behind anonymity at least affords you the luxury of comparing a known entity with a mythical one and still be able to slander your competitor.
So I ask again, where is the independent evidence to support any part of your claim of having a 99% hit rate and 0.1% miss rate?
I won’t expect to see any evidence so have no hesitation in stating your claims are not supported, substantiated or in any way real.
Kind of puts a different perspective on HIKs marketing when they are attacked by a manufacturer claiming they have something better but who can’t provide any evidence whatsoever to support the fake hype.
Using US terminology I’m calling BS on this one unless you can prove otherwise.
I'm not making claims, I am relating a personal anecdote about proper product development. Feel free to call BS. My contention is that an advanced Machine Learning analytic product should not be released until it works, and works better than existing technology, not trying to compare HIK to any other product. HIK's product doesn't work. If it makes you more comfortable, I will use the example of the watch I'm wearing. It was a gift from my wife. It worked on day one, and it works better than the product it replaced, which was me guessing at the time. The watch is very reasonably priced, but it's from my wife, and has a very person inscription on the back, which makes it priceless to me. Product created, and delivered to beyond expectations.
It isn't a dick measuring contest if you're the only one standing in the room, yelling at passers-by with you pants down.
My contention is simple. You choose to attack HIK within the same context as claiming mythical unproven statistics of your own product. Without any substantiation whatsoever.
My annecdote would be that whatever you claim to have access to I have something better. Indeed, your 99% is half baked as mine is 100% snd never misses anything.
So within your context, my product is far better than your half baked effort. That’s exactly how, as s manufacturer, you have presented your case.
So back to square one - prove it. As for dick measuring - I would suggest standing from the roof top shreaking mythical results whilst calling HIK half baked is indeed dick measuring, only without getting it out of your trousers.
Yes I do realize that. I have posted within the context that you are a manufacturer yourself and pushing a product (albeit someone else's) based on non-evidential facts. My gripe is that is so so so easy for anyone to take a pop at Hik saying its a half baked solution when no-one else has anything at that price point even available on the sleeves - and yet you feel you can still criticise despite having nothing better yourselves.
I do believe Hik have probably come to market too soon....maybe. But at least they've come to the market with something that everyone agrees is a big improvement on conventional analytics - just not as good as the marketing would suggest. Jesus christ are they the only company in history to have done this - I don't so. What I do know is competitors saying they have something better but...err.....i can neither prove it or demonstrate it and then gon to say Hik is "half baked" is not a credible or constructive comment.
Competitors are now playing catch-up. IPVM can think what it wants, but an improved firmware platform is imminent (as Ethan says), by which time competitors will still have nothing on the shelf. Just an army of denogrators throwing stones at glasshouses.
Yes the unit got some results wrong - probably a mix of incorrect set up and a firmware that has been improved since (in the EU). But is the unit so bad that IPVM have to lead with emotive words and phrases like "tested terribly", "serious issues" or is the issue the marketing hype? IPVM will say both - when in actual fact the marketing is the real issue and its no surprise to anyone that marketing always sets a level of expectation that reality finds hard to meet.
But at least they've come to the market with something that everyone agrees is a big improvement on conventional analytics
What is the "big improvement"? DeepInMind did not seem to eliminate false alarms, had unreliable object classifications, and was poorly supported by the manufacturer. What exactly did Hik improve on as it relates to current analytics?
Competitors are now playing catch-up.
Sorry, no. Hik is the one playing catch-up here, and quite poorly at that.
by which time competitors will still have nothing on the shelf
If you are referring to shelves at ADI that may be true, but plenty of companies like Avigilon, AgentVI, Jemez, VCA, and others, are shipping pretty good analytics right now, and have been for years.
UM15, Mr. Pickles kindly pointed out on Friday that I was the only one that doesnt read and retain the entirety of this discussion. He's got it covered.
To start the weekend lets take a look at UK "Piggate".
The infamous story was concocted by an adversary to PM Cameron to discredit him. The clever part is that everyone knew there was not a shred of evidence or truth in the story - BUT to force a serving Prime Minister into a situation where he had to deny having sex with a pig was damaging. A very clever piece of fake news that worked.
Now lets say John, the adversary publishes a negative story about Hik. John can say anything he wants and whatever Hik respond with would be defensive and damaging to the brand - regardless of the relevance or truth of the original article.
All Hik can do, wisely, is to ignore Johns attempts to force comment on anything he chooses to focus on. That is a good move....no matter how John hates it when they don't take the bait. Occasionally Hik will debunk the "blogger" - but sadly John only see's this as another excuse.
John has deployed this clever tool before in "Toiletgate", where he actively sought to damage Hikvision by inferring they must in some way be complicit in a installer fitting a Hik camera in a toilet. John never proved or attempted to show that in no way shape or form was Hikvision responsible, complicit or condoning the installation - but the damage was done. Hik were associated with a dubious installation of a camera in a schools toilet.
The tactics of fake news, tabloid reporting and attacking a leading brand are not unique to IPVM and are common place upon those seeking to gain commercially from the process. What John is doing is admirable to anyone seeking to increase subscription based income and page reads. It has less to do with personal development, understanding and dedication to objective information of the IP video market that many would wish. Derogatory comments directed at the Chinese nation would not seem to fit in with any part of IPVM's stated aims (and no, continual Chinese government conspiracy theories do not qualify as news). If you want proven state hacking news try reading your own press based upon proven fact.
Have a great weekend and don't let Piggate change your view of the mornings bacon.
you can continue to create straw men and knock the hay out of them, but stringing together words in a cogent manner does not actually mean that these straw men help or support the argument that you are trying to make.
Pragmatically, I believe that if the DiM NVR provides an improvement overall against non-AI unit's, then integrators will see this as a success and largely disregard advertising hype.
I read a comment somewhere that stated "....most of all, it does not really matter how well OV works relative to other analytics. As long as it is good enough, the ease of adding analytics and its Performing Analytics in your DVR distribution with your existing system will be sufficient."
Although describing the early days of analytics and DVR's, I believe the author hit the nail on the head and the performance of the Hik DiM NVR will indeed be sufficient as integrators understand that imminent and future firmware updates will only improve the product.
Clearly the Hik USA marketing needs shaping up - but this does not detract from the DiM NVR being a good product - destined only to get better.
Oh yes....the author by the way was a John Honovich.
if the DiM NVR provides an improvement overall against non-AI unit's then integrators will see this as a success
Sure, I'd go further. If DiM is 'just' equal to Avigilon at the DiM's price points, integrators will be excited and Hikvision will do well. That's a big if though...
this does not detract from the DiM NVR being a good product
And your evidence that DiM is a good product? The only partial positive response is DVS Dave saying it still has false alarms just not as bad as conventional motion detectors. That's still problematic for most and does not even get into the various human / vehicle / gender / face issues the unit has.
We will see where the firmware updates take DiM. However, they almost certainly need better hardware, one outdated Nvidia chip for 16 channels is going to handicap them regardless of firmware.
Second, multiple sources from Hikvision and close to Hikvision inform us that new firmware is expected to be available by the end of the quarter (March). We will check performance when this firmware is publicly available.
Finally, speaking with tech support now, unlike our initial conversations, the agent we spoke to was familiar with DeepinMind, asking several questions about configuration when asked about false alerts, and asking if it was windy in our test scene. Support also said that they were not directly familiar with DeepinMind and did not have a unit themselves, but instead used a "virtual" emulation of the GUI.
The analytical capability claims have always been over hyped and unfairly advertised by HikVision this is nothing new and neither is the age old excuse of a new firmware in the pipeline.
Line Crossing? Intrusion Box? yeah sure these have been used on large projects with the apparent capability of Static to PTZ linkage for smart tracking to take over. Did it work? NO, it was as useful as a chocolate teapot. The equipment performed so poorly that the return to site ratio was so excessive the installation company lost money.
Resolve: Good old fashioned IR quad beams and high quality PIRs (Redwall) hardwaired into alarm inputs.
NVR: 9600 series.
HDD: Full rack of 16x 4Tb HDD drives
Firmware Upgrades to resolve: Approx 9 (6 Standard FW and 3 specially written by HikVision to resolve issues)
HikVision need to thoroughly Beta Test their products before release and think more about the installer/end user relationships that can be damaged through poor performance of equipment.
A question for you undisclosed #18 as you are an end user. Do you think that Hikvision is so strong in residential / SMB because those features are rarely used? Those seem like features that would be targetted towards higher profile clients and the fact that they're writing custom firmware for you makes me think you are a larger scale client.
My inquiry is more towards whether they really have a place in larger scale projects.
Yes, we are trying to buy an authorized US version but so far seems to be delays or not available yet for that version. It is a priority for us since Dahua is marketing AI heavily.
