Month: April 2012

Late this afternoon, I happened to do a Google search for something I had written on my blog last year. The article came up in the search results, but when I clicked on it, I was redirected to a different site (a .ru domain). The target page didn’t load, though.

Primary investigative steps:

1. Try it again. Still happens.
2. Try search in a different browser. Still happens.
3. Shit, I’ve been hacked.
4. ssh to my server. Because I use Movable Type as my blog software, there are physical HTML files on disk for all my blog entries. Looking at the source code that should have been served up showed nothing unusual.
5. Examine the .htaccess file for my site. This is where the damage was being done. The .htaccess file controls things like redirects, and a bunch of code had been added to mine. Interestingly, it was checking the referer on each incoming HTTP request, and only redirecting viewers that had come from a search engine. If you typed “http://sunpig.com/martin” directly into the address bar of your browser, or if you arrived via a bookmark or a non-search engine link, you would be let in as normal. The hack was designed purely to bleed off search engine traffic.

Next step: find out when the damage was done, and how long this had been going on. The modified time on the .htaccess file was very recent – just an hour previously, in fact. It seemed unlikely that I had caught the hack so quickly after it had happened.

I logged in to my host’s control panel, and checked the account access logs. These logs showed that no-one but me had logged in to my account using ssh or the control panel in the last month. Good. I changed my account password anyway. I also notified my host, and told them what I’d found so far.

I used the unix find command to see what files had been changed in the last 7 days:

find . -mtime -7

This showed a bunch of files I knew I had changed or added myself, some log files that I would have expected to be changed, and also: a stack of .htaccess files where there should not have been any, a bunch of unfamiliar PHP files all called “pagenews.php”, and two “index.php” files that I shoud not have been altered in the last week.

Next, I identified all .htaccess files in my account, and all files called pagenews.php:

find . -name ".htaccess"
find . -name "pagenews.php"

Then I looked for common text signatures in the files, to see if there was anything else I was missing:

grep -r "on\.ru" .
grep -r "FilesMan" .

Okay, infection identified. I could clean up the affected files, but I still didn’t know how the attacker got in in the first place. Without knowing that, there would be nothing to stop them getting straight back in again.

The time stamps on all the affected files went back three days. Some were stamped today, some were stamped yesterday; only one dated back to Monday. I checked the Apache log files of web traffic, and found that yesterday’s time stamps matched up with unusual HTTP POST requests to the two index.php and pagenews.php files. Those files used some kind of obfuscation, so I couldn’t figure out what they were actually doing; but the fact that the file timestamps matched the web access logs, it seems like a reasonable assumption that those POST requests were actually writing files on my server.

However, the one index.php file with a timestamp of Monday didn’t have a matching entry in the HTTP logs. I checked the file permissions, and found that they were set to 666: readable and writeable by everyone on the server.

So my working theory was: at some point on Monday, a process owned by some other user on the same server process on the shared server discovered that I had an index.php file ripe for taking over. It injected the malicious code, but didn’t do anything else immediately. Then, on Tuesday, some other part of the attack kicked in, and started making HTTP requests to the infected PHP file. Because the affected PHP code is running under my account now, it’s free to muck around with other files that belong to me. So the infection spreads to other areas around my server…

Recovery steps:

• Remove the newly created pagenews.php files. Manually remove the infection code from the index.php files, and the .htaccess files. (The .htaccess files were modified, not overwritten. The malicious code was added to the start and end end of the file.)
• Lock down permissions on all files and folders in my account, so that no-one else on the shared server has permission to write to them.
• Remove unused code (old versions of Movable Type, Thinkup, lessn, inactive dev sites) to minimize attack surface for the future.

Upgrade to latest version of Movable Type (5.13)

To recursively apply 755 permissions to directories, and 644 permissions (read/write by me, read-only by others) to files:

find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;

Steps for the future: run a scheduled backup job for static files on the server. I already use autoMySQLBackup for daily backups of the databases on the server, but clearly I need to consider the static files, too. Vasilis van Gemert has an example here: https://gist.github.com/2415901.

Lessons learned:

• If you’re running on a shared server, make sure that your files are not writeable by others on that server.
• Backups. It’s not a matter if if something goes wrong, it’s a matter of when. My home backup strategy is pretty solid; my server backups are still lacking.

I just finished reading Kate Grifin’s The Midnight Mayor, and last week I read Christopher Fowler’s Bryant & May Off The Rails. Both books are love letters to London. They revel in the thick layers of history, above ground and below. The city is a living thing, metaphorically for Arthur Bryant and John May, and literally for Matthew Swift, the protagonist of Kate Griffin’s series. In both cases, the city can be angered or appeased, coaxed and cajoled into giving up its secrets. Bryant and May, detectives, discover a vital clue in the different patterns of upholstery used on the Underground’s 12 lines; Matthew Swift, a sorcerer, uses the Underground’s terms and conditions of carriage as a powerful magical ward to defend himself.

I’ve never lived in London, only visited, and so I only know it through the eyes of a tourist. But my most vivid memories of the city are of walking through it, not of the shopping or glitzy attractions.

Walking around Covent Garden when Abi and I took the train down from Edinburgh on day in the late 90s, just to have lunch at Belgo’s, and coffee with James. Walking from my hotel near Victoria to the QEII conference centre in the mornings and back again in the evenings, in June 2006 for the @Media conference; a steady soundtrack of At War With The Mystics by The Flaming Lips on my iPod. Walking from Waterloo to the Tower with Abi & the kids, and Jules & Becca; deciding that we were too tired to visit, so camping out at a nearby Starbucks for a cool frappucino instead. Walking from Victoria to Southwark last September for lunch with Bora, because it was a glorious day, and I had the time; gazing up in awe at the Shard under construction.

I’m more than a little tempted to plan a holiday in London solely for the purpose of walking the city, North to South, East to West. Not planning for any stops along the way; just taking it as it comes. Getting underway before dawn, and watching the city come to life around me. Lunch from a sandwich shop, dinner from a chippie. I don’t know how far I’d get, or what I’d see; I don’t actually know the city that well; but that’s part of the point. To walk, to see, to be.

For bookmarks, I like using markers of a time and place: bus, train, and plane tickets; cinema or concert tickets; significant receipts, especially from holidays; appointment cards; other people’s business cards, just after I met them.

After finishing the book, I leave the marker in the book for my future self to discover, years later, and to remember the circumstance of the journey or meeting. Sometimes fondly, sometimes not at all.