Blog

When the Good Guys get it Wrong

Recently, one of the websites I was working on had been flagged as a distributor of malware. Fearing the worst, I immediately checked the site for the most popular and obvious cause: a hidden <iframe> tag. My search bore no fruit however, and I began to worry that the issue was much more serious.

I followed the malware warning page link which led me to the StopBadware.org website, which then informed me the warning was initiated by Google.

Now Google does, I feel, have a right to warn about suspected malware infections on a website, and it does have its Webmaster tools to help locate the source of the problem. Unfortunately in my case it didn't work. All Google was able to tell me was that 2 pages (of just under 100) were apparently infected, although it wasn't able to tell me which pages.

I went over that code with a fine-toothed comb, checking every bit of PHP code, scrutinising all the Javascript, and looking at every line of the .htaccess file. I found nothing. No hidden frames, no inserted eval() statements and no redirects in the Apache .htaccess file. I used wget to download the entire site and then ran through that to find any suspicious keywords, which again found nothing.

At this point I was obviously worried. Was it that the site had been falsely flagged, or was there a serious problem that I couldn't find? I ran the site against several on-line security checkers, including Unmask Parasites, The AVG scanner and the on-line scanner from Mcaffee. When all of these came back negative, I was confident I couldn't find the issue because there was no issue to be found.

Throughout all of this I made a site review request to Google, mentioning my findings. That's actually quite a lot of requests, because the whole ordeal lasted just over a week. There was one brief respite during that period, but it was only a temporary thing that lasted barely half a day. No amount of rolling back code or perusing the FTP logs yielded a different result until suddenly, Google gave the website a green light. Whether this was because the automatic system was finally updated to ignore this false positive, or there was an element of human intervention is unknown, but I really hope it was the former to prevent this from happening again.

It's quite unfortunate really, that Google was in a position to have such an impact on this website. It has tarnished the name of the company whose site it was. The website is a key element to their business, so having it effectively down for a week is not good news. What is even more shocking is that Google is actually in the position to do this to a company, unrestricted and unfettered. There's no way to get in touch with a real person at Google to resolve such a situation, nor do they respond to emails about it (I sent a fair share, none of which have had any reply nearly two weeks later.) The matter was compounded further by the fact that StopBadware (the organisation which provides the malware domain lists to browsers such as Fx, Safari and IE) seems to trust Google wholeheartedly. I'm not saying that Google is untrustworthy, but it can, and does, get things wrong sometimes, and there seems to be no protection for the little guys when it does screw things up.

So, what can you do if you ever find yourself in this sort of situation?

• Put your website into maintenance mode. Leave a brief message about the site being down, and restore from the last good backup. If the site relies on very current data that hasn't yet been backed up, see if there's any way to restore the site without this, or at least check it out before it's put back on-line.
• Change all the FTP passwords for the site. If it has been attacked, then it means there's a chance your passwords could have been compromised.
• Check the site out. If you have shell access to the machine, then run find . | xargs grep 'bad-keyword' -sl in the root to determine if any files contain suspected malicious keywords, such as eval, iframe and maybe the name of the suspected infection as reported by Google or any other agent. Otherwise, download the site with something like wget -m yourdomain.com and then run the find line.
• Continually request website reviews from Google in the Webmaster tools area, and also StopBadware. Leave notes with your request to document what you have done.
• Email Google. You likely won't receive a response, but that doesn't mean they won't read it.
• If you have specific information on the infection, search on-line to see if other people have resolved their issues. If it is a genuine infection (which it will be in the majority of cases) then you should attempt to remove it as soon as possible.
• If there is an infection in the database and you would normally access the database through a web-based tool such as phpMyAdmin, then try using a browser that has no malware check to access this tool. It helps to do this from a Linux machine, just in-case there was an infection which will most likely be targeted at a Windows machine. I've found Konqueror very capable in this situation.
• Keep a record of every action you take. It helps when adding notes to a site review, and if you manage to solve the issue, your notes could help someone else who shares the same problem.
• Lastly, make sure all the systems on the website are up-to-date and secure. Unpatched forum software could be as likely a doorway to a hacker as a compromised FTP password.

It's a shame that this situation ever occurred, but even more of a shame that it could be allowed to occur. If nowhere else, surely the Internet is the one place where we should know not to have a single point of failure for anything. The very architecture of the 'net itself is a reminder of why this is so, and I think it would be good for us all to remember that relying too heavily on a single company or service for anything can have disastrous repercussions when things go wrong.