Late this afternoon, I happened to do a Google search for something I had written on my blog last year. The article came up in the search results, but when I clicked on it, I was redirected to a different site (a .ru domain). The target page didn’t load, though.
Primary investigative steps:
- Try it again. Still happens.
- Try search in a different browser. Still happens.
- Shit, I’ve been hacked.
ssh to my server. Because I use Movable Type as my blog software, there are physical HTML files on disk for all my blog entries. Looking at the source code that should have been served up showed nothing unusual.- Examine the .htaccess file for my site. This is where the damage was being done. The .htaccess file controls things like redirects, and a bunch of code had been added to mine. Interestingly, it was checking the referer on each incoming HTTP request, and only redirecting viewers that had come from a search engine. If you typed “http://sunpig.com/martin” directly into the address bar of your browser, or if you arrived via a bookmark or a non-search engine link, you would be let in as normal. The hack was designed purely to bleed off search engine traffic.
Next step: find out when the damage was done, and how long this had been going on. The modified time on the .htaccess file was very recent – just an hour previously, in fact. It seemed unlikely that I had caught the hack so quickly after it had happened.
I logged in to my host’s control panel, and checked the account access logs. These logs showed that no-one but me had logged in to my account using ssh or the control panel in the last month. Good. I changed my account password anyway. I also notified my host, and told them what I’d found so far.
I used the unix find command to see what files had been changed in the last 7 days:
find . -mtime -7
This showed a bunch of files I knew I had changed or added myself, some log files that I would have expected to be changed, and also: a stack of .htaccess files where there should not have been any, a bunch of unfamiliar PHP files all called “pagenews.php”, and two “index.php” files that I shoud not have been altered in the last week.
Next, I identified all .htaccess files in my account, and all files called pagenews.php:
find . -name ".htaccess"
find . -name "pagenews.php"
Then I looked for common text signatures in the files, to see if there was anything else I was missing:
grep -r "on\.ru" .
grep -r "FilesMan" .
Okay, infection identified. I could clean up the affected files, but I still didn’t know how the attacker got in in the first place. Without knowing that, there would be nothing to stop them getting straight back in again.
The time stamps on all the affected files went back three days. Some were stamped today, some were stamped yesterday; only one dated back to Monday. I checked the Apache log files of web traffic, and found that yesterday’s time stamps matched up with unusual HTTP POST requests to the two index.php and pagenews.php files. Those files used some kind of obfuscation, so I couldn’t figure out what they were actually doing; but the fact that the file timestamps matched the web access logs, it seems like a reasonable assumption that those POST requests were actually writing files on my server.
However, the one index.php file with a timestamp of Monday didn’t have a matching entry in the HTTP logs. I checked the file permissions, and found that they were set to 666: readable and writeable by everyone on the server.
So my working theory was: at some point on Monday, a process owned by some other user on the same server process on the shared server discovered that I had an index.php file ripe for taking over. It injected the malicious code, but didn’t do anything else immediately. Then, on Tuesday, some other part of the attack kicked in, and started making HTTP requests to the infected PHP file. Because the affected PHP code is running under my account now, it’s free to muck around with other files that belong to me. So the infection spreads to other areas around my server…
Recovery steps:
- Remove the newly created pagenews.php files. Manually remove the infection code from the index.php files, and the .htaccess files. (The .htaccess files were modified, not overwritten. The malicious code was added to the start and end end of the file.)
- Lock down permissions on all files and folders in my account, so that no-one else on the shared server has permission to write to them.
- Remove unused code (old versions of Movable Type, Thinkup, lessn, inactive dev sites) to minimize attack surface for the future.
Upgrade to latest version of Movable Type (5.13)
To recursively apply 755 permissions to directories, and 644 permissions (read/write by me, read-only by others) to files:
find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;
Steps for the future: run a scheduled backup job for static files on the server. I already use autoMySQLBackup for daily backups of the databases on the server, but clearly I need to consider the static files, too. Vasilis van Gemert has an example here: https://gist.github.com/2415901.
Lessons learned:
- If you’re running on a shared server, make sure that your files are not writeable by others on that server.
- Backups. It’s not a matter if if something goes wrong, it’s a matter of when. My home backup strategy is pretty solid; my server backups are still lacking.
Garbage’s new album Not Your Kind Of People has seen a mixed reception: fans love it, but critics have generally gone “meh.” My own first impression of it was that even the heavy rock tracks like “Man On A Wire” felt pretty lightweight compared to their older material. It took me few listens to catch the flow of the album, and to get caught up in the hooks and grooves. There are some really good songs on there. “Blood For Poppies” has a bouncy sing-along chorus that belies its stark lyrics about war and battle. “Felt” has a throbbing drum and guitar backbone with the kind of melty vocals I find hard to resist. And “Automatic Systematic Habit” is a wicked synth-heavy opener that I would rate as one of their best ever songs, if it weren’t for how terrible and plain the first verse is compared to the tight interlocking slickness of the second.
