Patrick here, again. It’s Sunday morning in Brooklyn.

Last night, Hosting Matters set us up with a new server (larger and more powerful, and all to ourselves, I gather), and we uploaded the complete March 1 backup of our home directory. This is why there’s been a March 1 version of Making Light’s home page visible for the last twelve hours or so at the usual location. Up until a few minutes ago, this was a version of the front page that looked all right, except that links to individual-post pages-with-comments didn’t work, and the Sidelights and Particle sections were empty, along with every other sidebar section that works by pulling data in from a file external to the main index page.

We also uploaded the March 1 export from the site’s MySQL database, and while we slept, Hosting Matters verified that the database export seemed to be okay, and re-created the database and the user. They suggested that we do a full “rebuild” from inside Movable Type, which should recreate all the internal links.

I’ve just now been trying to do that. The first thing I tried was a full site rebuild, but it crapped out and reported “internal server error” somewhere in the midst of rebuilding the second hundred individual-post pages. (A full site rebuild does individual-post pages first, then monthly archive pages, then index pages last.) The next thing I tried was an “index pages only” rebuild, which yielded the version of the front page that’s now visible, with the entire center column empty. I then tried an “individual archive pages only” rebuild, which once again failed in the middle of the second hundred pages. Finally, I made another attempt at a full site rebuild, and it failed (“Internal Server Error – The server encountered an internal error or misconfiguration and was unable to complete your request”) just after the 300-pages-rebuilt mark.

Rebuilds that fail to complete have been a constant bane of our Movable Type installation, but in the past we’ve dealt with them by simply trying again another day. Does anyone have a better idea that will get the site (at least, the March 1 instantiation of the site) up and functional now?

(Noted and of interest: You can get at the individual archive pages that are linked from the “recent comments” sidebar; you can even load the “last 4000 comments” page and get at lots and lots of individual archive pages from that. But as far as I know, until I get a full rebuild to happen, the middle column of the front page is going to stay empty.)

Hi, everybody. Teresa here. You may have already read earlier, incomplete versions of this post, which I published prematurely in order to get the THINGS TO DO RIGHT NOW part out fast. I said in them that you’d know you’d seen the complete version when the last word in it was dinosaur. That word is there now.

Onward.

First: apologies if you’ve written to us and haven’t heard back. We’ve been wrestling with the task of getting the pre-March 2008 part of Making Light back online. For a bit there, it looked like we might have lost the whole thing, not just March and April. That was exciting. Fortunately, Patrick found his alternate backup of the MySQL database that’s the difference between losing two months and losing seven years. As Erik Olson said at that point,

Good. We’re now at “this really sucks.”
Not “well, fuck, Maude, better sell the cattle.”

Thanks to many people who shall receive fulsome thanks once this settles down, we’ve reconstituted all the front-page entries, plus Sidelights and Particles. What we don’t have all of are the comments.

THIS IS IMPORTANT: collect your own comments NOW from Google or other search engines, then help collect others. Do it as soon as you can, because Google has already overwritten some caches with versions that end at 01 March. If you find this has happened to the Google cache of your own comments–for instance, Xopher, Niall McAuley, CHip, and NC Hanger have already gotten nailed–try the caches at Yahoo, MSN Live Search, or other search engines.

If you’ve been reading the comments at Making Light via RSS feed, please check and see what you have cached. It’s possible you have the only surviving version of some of our missing comments. We’re particularly on the prowl for new comments that got posted to old threads. We know to look for complete runs of comments from (f.i.) “All come singing” or “The rather difficult font game”; what’s far less obvious is the need to collect recent runs of comments in old threads like “Introduction to New Magics“.

Abi’s chart of what we do and don’t have isn’t up to date. That’s because Abi went to sleep (she’s six time zones east of Plymouth Rock), and Patrick and I are still figuring out exactly what we have on hand. She’ll be gone all day tomorrow, but we figure we’ll have the chart updated by tomorrow morning EST. If you’re doing intensive work salvaging comments and need more up-to-date information, send us your email address and we’ll try to reset the permissions so you can see our working spreadsheet in Google docs.

I’ve finally gotten in touch with Jim Macdonald. He’d been away most of today, and had no idea what was going on. The bits of Making Light’s database we have least hope of recovering are unfinished articles the editors had saved as draft posts in Movable Type. We’ve all lost some, but Jim had been working on the big final post in his Trauma and You series. He took the news very calmly, though it’s possible he was simply too tired to get upset. He says he’ll just have to rewrite the article from memory:

I’d gotten through traumatic amputations, degloving, and avulsions, and was about to start on incisions and lacerations.

When last seen, Jim was running searches on other people’s names. His suggested search string is “Comments posted to Making Light by [name of poster]”. If that doesn’t work, try something else. If that works, come back and tell us what it was.

(UPDATE: Jim has done as much comment-salvaging as he can do tonight. In the comment thread for this entry you’ll find his list of people he knows to have posted comments during the lost months, but whose body of comments he hasn’t salvaged. If you have time, please consider running some of those searches. If you know of other names, post a list of your own–Jim’s list is by no means complete. While you may wind up duplicating someone else’s efforts, you may also save the comments of people who haven’t gotten the word in time to do it themselves. Also, read the whole thread. There’s useful information in it. End of update: tnh, 0300 EDT.)

Salvaging the data is only the first part of the project. Once we’ve collected it, it’ll have to be reprocessed into proper MySQL format and grafted back onto the main database. We’ve had several offers of help, but if you want to add to them, please feel free.

…

In the midst of all this effort to salvage the missing months, we’re feeling awed and humble about the amount of help we’re getting. We’ve said all along that its readers are the best thing about Making Light, but that’s never seemed more true than it does right now.

(“Hey, look! It’s a dinosaur!”)

[Patrick here, not Abi.]

As luck would have it, I had to run off to band practice very shortly after the crisis began. I’m back. And I’ve had a life-giving, brain-restoring sandwich. As soon as I post this I’m going to start reading the email, and comments below, from our incredibly generous and helpful readers.

As mentioned previously, we do have a complete backup, downloaded with wget, of the site as of March 1, 2008. Hosting Matters has now set up new server space for us, and presently I’ll be trying to upload the backup; if everything works, Making-Light-as-of-March-1 will appear at the usual URL. Much more likely, however, is that I’ll have a zillion finicky permission problems that I’ll need someone’s help with before we get to that point.

At this point we have HTML of all the missing posts, and quite a lot of the missing comments, but many more comments remain to be excavated. Please, if you have been using an RSS reader to follow ML comments–generally or in specific threads–consider that you may actually have copies of comments from the last two months in your reader’s cache folders. Please take a look.

More to come. My actual suspicion is that once we get the March 1 version of ML restored, the best way to re-attach the missing material is going to be by editing and manipulating the MySQL database in the guts of the site. I know very little about this, and the small amount that’s been explained to me by helpful readers is pretty much worn away by time, but it seems to me that in principle it should be possible to automate, or at least semi-automate, the processes of:

(1) stripping away extraneous cruft from various saves of ML content in HTML and XML form, and–

(2) –wedging it all into the appropriate fields and tables in the MySQL database, so that–

(3) –all of ML reappears, right up to the present, along with all the–

[…]books we bought in college and sold for half-price unread
And sacks and sacks of earring backs lost under someone’s bed
And baseball cards and army men and model planes galore
And every tiny plastic high-heel Barbie ever wore

(Thank you, Austin Lounge Lizards. Music here, if you’re willing to sign up for a free trial of Rhapsody.)

Of course, knowing that something is theoretically possible doesn’t mean it’s practical, but I bring it up for discussion nonetheless.

Below is an initial list of postings that have been active recently, to the best of my memory. I have noted caches that I have URLs for, kindly provided by people in these threads. I am digging through my own caches. If you have better caches of data, please send them to me (abi at the domain we’re on), Patrick, and Teresa (their initials at panix.com). We will update this as we go.

(Why, yes, there is a spreadsheet!)

Making Light

Particles

Teresa has been looking at this and will report

Sidelights

Vide supra

Nielsenhayden.com

Placeholder for anything from this page, in particular the new narcolepsy postings

See Patrick’s LiveJournal post for details, but the short version is that we’ve lost everything from March 1.

This thread is for co-ordinating solutions to the problem. What we need:

1. Any caches of Making Light since March 1. If you have tabs open with ML, please save it, and send to Patrick and Teresa (their initials @panix.com), please.
2. Any other saved versions: please post what you have here
3. Offers of assistance in processing these saves
4. Wisdom in the ways of LJ feeds: apparently there is a makinglight one, which has data back to April 13th
5. Movable Type and blogging gods and gurus: suggestions, ideas
6. Time machines you can loan us
7. Suppliers for magical pixie dust

See Patrick’s LiveJournal post for details, but the short version is that we’ve lost everything from March 1.

So come on over here and put your feet up. I haven’t any super-linked quotes to start it off. Suggest one, if you will.

Welcome, anyone who wants to park here while Patrick and Teresa’s living room is under tarps. The beer is in the bathtub, the chips are really crisps, and I might bake a cake later.

In the meantime,

Otium, Catulle, tibi molestum est
otium exsultas nimiumque gestis
otium et reges prius et beatas
perdidit urbes

Translate, mangle, scan, discuss, ignore.

Note for my usual readers: the Moveable Type installation on the site where I hang out rather a lot is broken. I’ve invited the community over here until it’s fixed.

Don’t let this stop you from commenting – these are great folks.