Denial Of Service Attack On IPVM

At ~4pm ET, IPVM started to be hit with a DDoS attack. The attack is still ongoing.

We have implemented a number of improvements and have some more coming shortly.

Apologies for problems using IPVM today and we will debrief this, in more detail, once everything is back up 100% and can do a postmortem. Thanks!

[Update: Everything is back up 100%, we will add more details after we review them today.]

[Update 7/9/2020 Attacks have restarted and there have been some interruptions though our infrastructure is substantially improved and continuing to be improved.]

[Update 7/10/2020 Attacks stopped and infrastructure is further improved.]

[Update 7/10/2020 4:30pm Another improvement of infrastructure is in place and not only are we much better defended but the site is loading much faster.]

[Update 7/10/2020 6:00pm A third attack was launched, with moderate short term impact but the attack has now been defeated. Unclear when or if the next attack will occur.]


can the postmortem tell what type of devices were used in this DDOS attack?

the Mirai botnet attack hit (primarily) east coast US DNS servers... which meant lots of sites were unreachable.

from what I can tell at least from my machine (and others I have access to around the US), IPVM is the only site I am having issues with connecting to (from all locations) - which would lead me to believe that this attack is pointed at the IPVM hosting servers directly.

calling bashis, come in bashis...

Logged in a few hours ago and when I used the profile link got after a while an error up.System was very slow so I logged out

Chinese attacking IPVM

HIK is behind attack:)

Oho! Some computer science engineers have been upset with the feedback on their efforts to make yet another market-disruptive video startup?

Looking forward to see who's behind this... Guess we'll never know anyways.

"Guess we'll never know anyway"

Are you joking?

JH knows everything:)

I'm interested in learning further about this DDoS attack. Subscribed.

Update: everything has been back up for ~12 hours. We will add more details later today after we review them. Thanks!

I feel your pain. I work for a firm that has been very critical of China and the CCP. Anytime we publish something, or my boss makes a TV appearance, we get slammed with DDoS attacks. Millions of hits per minute from IPs that ping from various eastern European countries...

If it's any consolation, if China's DDoS Army is attacking your site, it means you are doing something right... keep up the great work, John.

In fairness, we have angered enough people around the world, including America, that it could easily be any number of actors or even random. We will see what we can learn, though, the focus for us is strengthening our infrastructure to handle attacks, regardless of source or motivation.

I’m sure it was just a random attack and had nothing to do with bad reviews, exposing ownership by foreign countries, being quoted on TV and Newsprint trashing products and sellers. What’s not to love about you guys?

And here I thought my work had just decided to block IPVM., yesterday.

This is horrible news, but we are confident your team of experts will have it fixed.

We noticed a bit of data leaks as well regarding clients passwords from IPVM and Ill keep digging but it seems you may have gotten hacked not just a DDOS.

People dont just DDOS anymore, thats typically a distraction to force you to get your admins to login to your CPANEL that has already been infected. Is there a reason you have so many different domains out there.

We took a snapshot of a Darknet site with a list of hundreds of IPVM passwords and client data. Have you seen it? Go to Ghostbin and type your name in, also seems theres fucker out there that is buying all of the IPVM domains. Like Ipvm.us and IPVM.Blog, or did you buy them?IPVM Image

What are you doing to ensure that these sites dont steal or infiltrate your clients. This may not be a great day for my assurance in IPVM that it is prepared to keep us safe, everyone is getting hacked. This is why im doing this anonymous for my safety but I monitor these boards.

Wish you the best John. These hackers are nuts arent they.

I'm not really sure about the data on the "darknet" site (I'm pretty sure Ghostbin is not darknet because I can find it without using Tor). The screenshot says "Sourced from Exploit.in data". Exploit.in was the name of a large list (593 million) of email and password combinations that appeared on the Internet in 2016. It has nothing to do with IPVM directly. Rather, some other site (probably several other sites) had a data breach, and John and/or Ethan had an account on that site. So that part is not IPVM user data.

IPVM.us and IPVM.blog are interesting in a legal way. IPVM.us was registered by Jasun Tate of Black Alchemy Solutions Group, and also the CSO of X.Labs. Both domains were registered the same day (May 9, 2020), so it's likely they are both connected to X.Labs. Nothing has been done with them, however. If X.Labs tried to do anything malicious with those domains, it would be a simple legal matter to stop them. Note that I have seen no evidence yet to connect X.Labs to the DDoS or any other cyber attack.

You are correct about the DDoS being used as a distraction. This is called a "blinding" attack. The annoying part with DDoS, however, is that it's been commercialized. It's not hard to buy an hour of DDoS against somebody you don't like. That said, I don't know what kind of engineering is done on the backend. Depending on what IPVM has set up, a successful DDoS could have required a more sophisticated attack.

We have implemented measures to protect against the type of attack we experienced yesterday, and are currently reinforcing those measures. From our investigations so far, we've determined that over 6,000 globally distributed IPs sent millions of requests to the site, overloading the servers. The majority of these requests were made to a handful of pages, and we have no evidence of brute force login attempts or anything of the kind.

As far as protecting user data, we use industry best practices for encrypting and storing user information, such as logins. Concerning the results you found for John and Ethan's emails, they likely come from old data breaches on unrelated (non-IPVM) services, as listed by Have I Been Pwned? (which even lists the Exploit.in data set for John's email), as UI#6 mentioned.

We currently have no reason to believe any IPVM logins have been accessed, decrypted, or leaked. If you have reason to believe otherwise, please let us know.

You raise interesting points, from our research it seems like there was and is always room for improvement in companies that people rely on for expert opinion in security and matters of technology like IPVM. Especially one whereby people like us pay for services.

Great points, Im going to look into this Jason charachter. Black Alchemy what kind of name is that anyway haha wierdos :)

was IPVM attacked by IP cameras?

imo, any camera model which received a bad review here would have the means and motivation to attack and settle scores ;)

is my personal information safe with IPVM? I am beginning to feel like it may well not be?

You put personal information other than email and name? Seems silly for someone so concerned about privacy.

See Seth's comment above. At this time they have no reason to believe any IPVM logins have been accessed, decrypted, or leaked.

You've made it to the big show officially. No website is legit until they are dos'ed.

And here I thought getting blocked by China meant they were in the big show already.

Or getting sued by a manufacturer.... Or getting insulted by executives.... Or kicked out of a booth.... Or having somebody set up a blog (and several Twitter accounts) dedicated to mocking them...

Sheesh. How much does it take to be part of the big show?

How much does it take to be part of the big show?

Drunken fistfight in the parking lot after ISC West.

Goals.

Update 7/9/2020 Attacks have restarted and there have been some interruptions though our infrastructure is substantially improved and continuing to be improved.

FYI, the attacks are peaking at nearly 5 million requests per 5 minutes.

FYI, the attacks are peaking at nearly 5 million requests per 5 minutes.

traffic like that hasn't been seen around here since Canon bought Axis :)

Sigh, that’s like almost a million a minute.

Sigh, that’s like almost a million a minute.

on average

How long are these attacks lasting? I think this one started at 6pm, ET, but I didn't notice when it ended.

This attack ended between 11 pm and midnight yesterday, Thursday, July 9th.

FYI, your CloudFront settings may need to be tweaked. I'm missing a lot of images all over the site. All avatars are gone and also all images inside articles. Most of the article thumbnails are still there on the home page, but I'm guessing that's because of caching. Your newest article has no image for me.

Thanks for the information. At the risk of sounding exceptionally canned: while we have not directly observed these problems, we are seeing increased error rates from the changes in the last few days. We have updated our settings to mitigate this, and are closely monitoring the error rate to ensure it returns to normal as the changes take effect. Please let us know if this continues to be a problem.

Same for me, if I do a full refresh of the page then I lose all images.

Okay, images are back for me now

Very interesting timing of this. HIKVision has been known to carry these out against competitor sites as well as especially during RFP/RFQ periods of government bids. Most notable was in March 2020. With all the bad press this was their obvious next step.

We do not know who is behind the attacks. We are doing more investigation but the focus has been strengthening our defenses.

looking on the bright side, your web metrics must be thru the roof :)

Do you have proof of what happened in March or any other instance?

No, I do not work for Hikvision.

I just want to say that you guys are doing a great job - keep it up. Reacting to stuff like this is certainly a challenge, and unfortunately the attackers keep getting more clever. But as someone else mentioned above, it must mean that you're doing something right to garner this kind of attention.

IPVM will go offline within the next 15 minutes for about two hours to complete final upgrades to our infrastructure.

Update: Another improvement of infrastructure is in place (we finished significantly earlier than planned) and not only are we much better defended but the site is loading much faster.

Thanks to Duc and Seth!

It is a lot snappier I noticed immediately

We are under attack again. This time, so far, we are holding up better but how it evolves remains to be seen.

A third attack was launched, with moderate short term impact but the attack has now been defeated. Unclear when or if the next attack will occur.

Glad to hear it! Hopefully they'll get tired of attacking.

A fourth attack was launched this morning, with moderate short term impact. The attack is over, at least for the moment. We also have made further optimizations to strengthen our defense.

The peak size of this attack doubled to nearly 10 million requests per 5 minutes.

Amazon statistics show the peak bit rate was 216Gb/s and the largest DDoS attack against Amazon today.

That's a pretty impressive amount of data - these guys are really cranking things up.

I am curious to know what the costs are of doing it at this scale. I've read some articles online that talk about botnets for hire, e.g., recent Krebs article:

Some of the IoT botnets enslaved hundreds of thousands of hacked devices. For example, by November 2017, Masuta had infected an estimated 700,000 systems, allegedly allowing the defendants to launch crippling DDoS attacks capable of hurling 100 gigabits of junk data per second at targets — enough firepower to take down many large websites.

Anyone with any insights on the cost of doing this, please share or email me john@ipvm.com

I think it's fairly obvious which bad actor is behind this. Maybe not sanctioned by the companies themselves, but most certainly the party.

This is not a fault of the Republican Party.

Well, it appears one person saw the humor and one didn’t. Which is which, I will leave to your own thoughts.

A fifth DDoS attack, and the second today, is in progress currently.

The attack is over, likely the shortest and least successful to date. We will see what happens next.

This must be avery sophisticated attack. The timing of attacks is also worth noting. Evening/early morning. Time zones place this in optimal APAC/Far East hours.

The cost of such a high rate sophisticated attack is not cheap. Someone must have serious beef with IPVM to spend serious $$ in being successful to take the site down again and again.

Someone must have serious beef with IPVM

That strikes me as reasonable.

in being successful to take the site down again and again.

The general effectiveness is trending down. Let's see what they do next.

I would say threaten a lawsuit, but it’s already been done many times.

This must be avery sophisticated attack.

volume-based DDoS attacks are the opposite of sophisticated.

The cost of such a high rate sophisticated attack is not cheap.

exactly wrong again...

renting botnets for DDoS attacks has been a thing since 2006ish - and the costs have been negligible for at least a decade.

and the costs have been negligible for at least a decade.

I read that article as well, what is not clear to me is what size botnet you get for $67, you have any idea on that?

this story from Securelist a few years ago actually shows cost-breakdowns for size (in gbps) and types of DDoS attacks - like this one in Russian (I think):

IPVM Image

i don't speak russian, but even so i can spot a scam 10,000 miles away:

IPVM Image

i'd imagine that the maroon colored hat on the corner of the first offer says something like "Best Value or Recommended or Putin's Choice", however looking closer it appears to give you only 600 "CEKYHA" at $10 per "MEC", while the Orange plan is offering 1200 "CEKYHA" at $12 a "MEC".

600 more "CEKYHW" for only 2$ more per "MEC"?

Unless, the maroon hat actually says "Biggest Rip-off" or "CEKYHA" means lashes with a wet noodle, the green offer is woefully low on "CEKYHA".

Я говорю по русски

The maroon roughly means best offer.

секунд атаки = Seconds of attack

месяц = мес = Month

I think you got the idea.

From reviewing the IP addresses of devices attacking IPVM, the devices are primarily Mikrotik routers, per Shodan.io.

Here is a TrendMicro post on Mikrotik botnets.

Hi John and you all.

Thank you, this is a a freebie, good educational bonus to to the course just completed. Appreciated as it is very educational to me.

A sixth DDoS attack, and the 3rd today, is currently underway.

The sixth attack was a failure and has ceased. We also have further defensive options that we will implement if the attacker can make headway.

right now, getting every other page 403 forbidden, but when it does load, loads quickly.

hey, i'm sure you guys know what you are doing, but since it's a closed forum, couldn't you start by whitelisting known ips from members, and bounce everything else when trouble starts?

or are there too many requests to even filter by ip?

edit: or maybe they are spoofing enough ip's that they would eventually find a members ip and then distribute it to the bot net?

couldn't you start by whitelisting known ips from members, and bounce everything else when trouble starts?

We could theoretically but I don't think we need to do that. Also, it would still cause lots of issues anytime some member was coming from a new IP address (which would be frequent).

Also, it would still cause lots of issues anytime some member was coming from a new IP address (which would be frequent).

so the whitelist would be appended with an ip whenever a valid login occurred, during times the site wasn't under attack. meaning it would be up to date. you would only flip the bounce switch when under attack. only in that period would someone using a new ip be denied. which is no worse for them than before, and much better for everyone else.

another idea is to make a version of your home page with the absolute bare minimum elements. like user name and password only.

when you're under attack you switch to bare bones login to reduce the load.

if they have a valid paying login or two and use it to attack, you can see what account they're using and delete them.

3 of the 6 targeted attacks were on Saturday (today) - a day that sees the lowest engagement among IPVM subscribers compared to any other day - except maybe Sundays. (are multiple attacks coming tomorrow?)

it's almost as if the attackers have no understanding of the industry - nor how the industry uses the primary source of information in the industry.

Yeah, I've wondered about that too.

Is there some more immediate goal that we don't know about? Like maybe a particular country is doing a study on IoT devices and a manufacturer is trying to stop officials from that country getting access to IPVM's research. (If you're following the news, you can guess what I'm thinking of, but we really don't know who is behind this yet.)

Another option is that it's a simple punitive measure. I don't know how much the infrastructure improvements and general traffic cost, but it's probably not free. If it's an organization that has just had enough of IPVM, maybe they'll just keep mashing the DDoS button until they feel better.

it's almost as if the attackers have no understanding of the industry - nor how the industry uses the primary source of information in the industry.

Thanks for educating them -;

No attacks since yesterday.

JH,

Was attack against all IPVM web server IP's or just one?

just curios

They have attacked in different ways. We can tell they are trying to figure out what to do next as we improve our defenses. Because of that, I am not going to disclose or elaborate on what we are doing right now. They can spend more time and energy trying to figure that out on their own. Say hello to them for me, #3? :)

Update: no further attacks since Saturday's failed attempt. We remain alert and are continuing to upgrade our defensive and security efforts.

We are now being mass signed up to email newsletters:

IPVM Image

I don't know for sure if it's related but I am mostly bemused at this point....

Hackaday has some good content at times, you should hit "confirm" on that one :)

I'm waiting for Robert J Miller to wax prophetically on how sophisticated Subscription/Email Bombing attacks are.

"Historically, journalists have found themselves the target of email bombing campaigns in retribution for critical stories."

We are now being mass signed up to email newsletters...

I don't know for sure if it's related but I am mostly bemused at this point....

once you've been spammed by Longse, everything pales in comparison...

All this reminds my of The Cuckoo's Egg by Cliff Stoll - one of the best books I ever read.

IPVM Image

if you liked that book (which I also loved), try these:

The Watchman

IPVM Image

Takedown

IPVM Image

Ghost in the Wires

IPVM Image