MIR is no longer hosted on the Rackspace Cloud

For a good few years we’ve been hosting Mobile Industry Review (“MIR”) on the Rackspace Cloud.

I’ve loved it. The concept is this: You pay $100 a month minimum. That buys you 10,000 ‘compute cycles’ and a ton of diskspace and bandwidth. For a reasonable sized site like MIR with around 250,000 visitors a month, it’s rare to go over the 100 dollar amount. Cost, however, isn’t the key issue for me: It’s availability.

I absolutely loved finally being free of the hardware panic. With the Rackspace Cloud, you upload your code, files, everything to their main server. And your site is then served via high-availability sooper-dooper cluster. Super-shit-hot-wonderful-amazing stuff. That means that should the Rt. Hon. Scoble link to you — as he has done now and again — I don’t feel it, technically. Rackspace’s infrastructure scales dynamically and boom, 10,000 people are served instantly.

Yes you’ll burn through your ‘compute cycles’, bandwidth allotment and so on — but technically, you won’t have a problem.

It’s genius. Utter genius. And I just couldn’t get enough of it. Having hosted sites for years on dedicated servers, I always had to make sure I was near an SSH or Telnet connection (BlackBerry is your friend for this, by the way — the first Blue BlackBerry, the 7230, was it? Or 7320? That had telnet capabilities) so I could login and restart Apache (web server) or MySQL (database server) in case something had screwed up. Anyone who currently manages a dedicated server knows what I mean. Despite putting an inordinate amount of control measures and monitoring, you still have to keep on checking stuff. And then — usually when you’re on your way out to have a nice dinner — you’ll get a note from a client’s client’s mate’s friend’s budgie’s mate, to tell you that the site doesn’t seem to be responding. On closer investigation, you’ll find it’s not the 20-second fix you’d hoped for… and that… arse, yes, you will need to replace the hard disk and… shit, yes, despite doing backups hourly, you have lost 6 hours worth of data. Or you can recover that last 6 hours, just, it’ll take 11 hours. And the client’s screaming… and the site’s down. And you’re now getting complaint emails from end-users.

Arse. Arse and thrice arse.

I think we’ve had about five periods of sustained downtime, each no longer than about 30 minutes, in around a year. Usually to do with a DNS error or something like that. All resolved quickly. All painless. Utterly phenomenal.

But, here’s the bad news.

WordPress.

The content management system that runs MIR and a ton of other sites is susceptible to occasional ‘issues’. And if you haven’t set it up right — or if you’re running fast during setup — there are ample opportunities for hackers to inject malware into your site and boom, it’s a total arse to fix. The malware is more or less automated. Scans discover holes that allow systems to place control files on to your site with write permissions. Before you know it, every page of your site will contain hidden links to an array of porn and spam sites. Google is pretty good at tracking it and advising users not to visit your infected site. Annoying but useful.

So WordPress needs to be installed and configured correctly. And it needs to be regularly updated.

I’ve never had a real problem until a few months ago.

From the brief reading I’ve done, Rackspace suffered some kind of security issue at some point. Michael over at Smackdown has a good summary of his findings. The security issue resulted in a ton of my sites — and I host a lot of them — getting infected.

It’s been a total nightmare. The malware installs itself into a ton of different hard to find and hard to recognise locations across your filespace. So the only way to actually be sure it’s gone is to completely reinstall everything. From scratch. In some instances, the malware even infects your database.

I don’t have a lot of time to arse around with this kind of thing. I spent an afternoon recently looking into the scale of the damage. I did a good few hours looking to see if I could find any competitors to use for hosting.

In the end, I thought ‘screw it’.

And I spun up a Rackspace Cloud Server. (This is separate from a ‘Cloud Site’ — which is how I hosted MIR, Mobile Developer TV, and so on)

I configured the server. I stuck on the firewall, Apache, MySQL etc.

I transferred the sites.

And now we’re back on a dedicated server.

Yes I’m still doing business with Rackspace — I’m just hugely disappointed I’ve had to stop using their Cloud Sites system. I did my best to try and clear out the rubbish malware, but it was absolutely everywhere. And I no longer had the confidence in their ability to deliver decent service. Further, I didn’t want to make a new Cloud Sites account only to find it was hosted on the same compromised-but-not-but-still-dodgy server IP.

As for the rest of the dedicated server industry? Rubbish. I looked around for instant server activations. I can’t be spending days evaluating services and when I need technology infrastructure, I want it now, not at 9am the next morning once some bored customer service agent has finished having his morning coffee.

So I used my Rackspace account to create a dedicated server (instance). The one I chose is $0.12 per hour, plus bandwidth. That’ll do.

As a measure of speed, the server instance was live in about 45 seconds. Apache was installed by about 1 minute 20. Heh. And MIR was moved, in its entirety — lock, stock and DNS barrel — in about 3 hours.

This is the temporary solution. We’re live, we’re free from malware, we’re free from infection from the rest of the Cloud Sites. And I can get on with screaming about the mobile industry’s total lack of innovation, rather than messing around with firewall config scripts.

If you’ve any dedicated/cloud/hosted server suggestions for Mobile Industry Review and our band of sites, they’d be most welcome. I now have the patience to calmly evaluate things.

, , ,

  • “hackers to inject malware into your site and boom, it’s a total arse to fix.”

    Welcome to my world. That’s why last100.com is down and marked by google as malware. I’ve almost given up mid clean up. It’s not the first time but I’m on a shared server hosted my MT.

  • mvandemar

    I think your database suffered some funky character translations in the transition. Look at the titles in the sidebar that have a price in them and you should see what I mean. I can help fix that if you like.

    Also, if you want I could probably teach you in a couple hours how to find all infected files and records in a hacked installation quickly and easily (I wrote the post you linked to). I developed a recipe that I follow that is very thorough. Just let me know.

  • I need help with last100.com big time.

  • Ewan – Really sorry you experienced this – so did I an my account. I learned how to really set it up, got it locked down, then cleaned everything up. Painful, but nothing bad in a few months now. Of course, that is not good enough for us and we are working hard to make automated changes to our infrastructure that will help with some of these issues. We are looking into ways to harden not just WordPress installs, but all popular CMS apps. WordPress is the largest target, so the scripts attack it aggressively – and we fight back as hard as we can, but scripts are fast, and sometime there are many copies doing attacks at the same time – we do stop most of them

    I hope you will watch our progress on this over time – and perhaps come back to Sites when we've implemented changes. It does work extremely well for sites that need bursting capabilities.

    I don;t have any time-line – but I do know we are actively and urgently working on “lock-box” solutions.

    Thanks much,

    Rob La Gesse
    Chief Disruption Officer
    Rackspace Hosting
    210-845-4440
    rob@rackspace.org
    @kr8tr

  • Need some help with that Steve?

  • mvandemar

    By the way, I tried looking at that site to see if I could find contact info for you, but it is password protected.

Powered by WordPress. Designed by Woo Themes

Real Time Analytics