How I Went From 5 Users to 5000 Users Overnight
I run a small web hosting company called MagicEdit which lets web designers setup a CMS using standard HTML pages. It works great because it gives non-programmers an easy way to create websites that they can give to clients and their customers can edit right in their browser without having to mess with any kind of confusing backend.
Yesterday one of our customers complained that some of his sites were loading a little slowly. He had done some investigation and thought maybe there was a problem at Amazon EC2, where we run the servers. I poked around and couldn’t find any problems, I was able to ssh into the server just fine and typing was very responsive, and the CPU load on the server was very low, less than 1%. But I was able to repeat his problem and could see some pages were taking a little long to load. As part of troubleshooting I restarted Apache and that seemed to fix things so I thought maybe it was just a glitch and left it at that.
Went back in the logs as far back as I could to try to see when/why someone had started a botnet DDoS against us. Doing counts in the logs and there’s 50k different IPs per week and nearly a million requests for that script over the last month. That’s a pretty huge botnet! I really have no idea when it might have started since it wasn’t creating any real problems for the server so I hadn’t noticed it before. It’s a very small file so in terms of overall data use it also didn’t really make any difference that would stand out.
I took a look at how many simultaneous connections were open to Apache and was finding that there were 300-400 requests going on. Seemed like that might be the source of the slowdown, but what could I do about it? Since the script file is just some stuff that customers don’t access directly, I moved it to a new URL so that the old one would return a 404, to see if that would change anything. It didn’t, and now my log was just filling up with 404 entries.
With the script only being used by our system internally, I moved the whole site to a new subdomain and sent the old one to 127.0.0.1. But there were still tons of nameservers with the old IP cached so I was still getting hits although not as many. Then I started wondering if maybe someone was doing something causing real browsers to grab the script. Did some searches in google to see if I could find a site that had embedded our script but found nothing. Tried lots of different variations, and even tried Bing, but couldn’t get any results.
All the traffic was coming from thousands of subdomains off a couple of .com & .biz domains. For whatever reason though I can’t get a single one of those pages to load, it always errors out. Not sure if it’s because their Windoze server is overloaded dealing with so many spam clicks or if it’s doing something to prevent other traffic from loading the page or what. But I was able to a search for the domains and got some hits in Google and was able to pull up the cache and sure enough, there’s the link to our script!
CUT AND PASTE
Why the heck would someone embed our internal script on their web site? The script is just used by our web site, and it’s only used when a customer has logged in on their own site and wants to make edits. The script is hardcoded to work with our CSS and tags, and really woudn’t be of much use to anyone else. And in order for them to use it, they’d have to make their own copy and modify it, so it makes absolutely no sense for them to link to it on our server.
All I can figure is that whoever made the spammer’s site googled for something and ended up on one of our blog posts that talks about how to create a photo gallery or validate a form or something. They signed up a MagicEdit account and installed our template, and then did a view source on the page and cut & pasted it instead of opening the zip file on their computer. Since they were logged in, they got the script tag that was injected in on the server side. With the script being harcoded for our site, it probably didn’t do anything on their site and they didn’t notice it.
WHAT TROUBLE CAN I CAUSE
Well now that I have a huge source of traffic, what can I do with it? I expect the spammer will get his site fixed soon, so it’s not going to really do much for me. I also don’t think it’s a good idea to send the traffic to any of my sites that are running ads, since about all I know is there’s some kind of spam and/or affiliate marketing scheme going on and I really don’t want my ad account getting banned. Well, why not just send the traffic to MagicEdit where they pulled the script from?
I let the redirect to MagicEdit run overnight and woke up to find a ton of free account signups. I really doubt they knew what they signed up for though since these people are clicking from a spam site. I’m sure they were confused and were just filling out forms hoping it would get them whatever they’re looking for.
Another site I have is Wishzilla, which is a great way to share gift ideas and manage a wishlist. It’s very popular with moms and grandmas because it lets them know just what to get for their entire family and without spoiling any surprises. Plus, it’s getting close to the holiday season, seems like a perfect place to send people!
For fun I pulled up Wishzilla in the analytics so I can easily see what kind of traffic I’m getting. Oddly, all of the traffic is coming from the US, and probably around 90% of it is coming from the southern US. Very strange. I would expect there to be gullible people world wide that would be hitting a spammer’s site.
We’re also getting the occasional support ticket through Wishzilla complaining that they can’t get to the website they wanted to go to, or that their “customers” can’t get to their website. Not sure if there’s some affiliate marketing thing going on or what. I’ve even had a few complain that they can’t go to any website now. I think the spammer might have had them install a toolbar which is loading our script! What a stupid spammer!
I’m still surprised that the spammer hasn’t fixed their site yet. I guess this was just a set it and forget it deal and they’re too busy with the next scam they’re running? I’ll probably never know what’s really going on, but this has sure made for an interesting tale!