Tag Archives: work

Entomosemantics, or, how to talk about bugs

One of the skills they pay me the big bucks medium-sized Euro for at work is assessing the risks of changes going into production. To do it, I’ve become pretty good at evaluating the system that is being changed.

I could snow you with talk of checklists, metrics, and charts, but really, my most valuable analytical tools are my pattern-matching wetware and my experience. With those two things, I can usually describe the current state of the system and estimate its chances of going horribly wrong in the near future, just based on gut feel.

Below are my private terms for the various states of computer system health. I use different ones in official reporting. Usually.

  • clean: The system runs smoothly, with no visible bugs. I read the logs to calm down after stressful meetings.
  • stable: There are the occasional interface bugs, but the thing runs reliably. It feels like a melon you tap in the supermarket and decide to buy.
  • scruffy: Most users hit some kind of bug or another, but they can make it work most of the time. Regular users have workarounds the way commuters have rat-runs that avoid traffic blackspots.
  • buggy: This is when users begin to see the bugs they encounter as a pattern rather than individual occurrences. They start to wonder if the pattern of bugs indicates a deeper unreliabilty. They’re right to.
  • brittle: Bugs aside, it pretty much works…right up to the point where it shatters into little tiny pieces.
  • fragile: It falls over a lot. Ops can pretty much always get it back up again in a reasonable time. We spend a lot of time apologizing.
  • fucked: It’s broken. Again. Fortunately, we have backups, and we’re fairly sure they’ll work.
  • comprehensively fucked: The backups didn’t work. Shark time.

Entropy tells us that, barring intervention, systems tend to move down this sequence. But it’s not a linear progression. For instance, brittle and fragile, are parallel routes to fuckedness. They’re basically two different failure modes: the Big Bad Bang and Death by a Thousand Cuts.

The applicability of these categories to other matters is left as an exercise for the reader.

Cross-posted on Making Light, where any comments will live.

How to sour a community, in one easy lesson

Simple. Tell them that they’re not one.

It won’t destroy it, of course. Wherever a group of people collaborate for a common endeavor, there we find community.

But communities come in different flavors. My favorite kind includes a substantial amount of trust among the community members, and between them and their leaders/moderators. They are often powerfully goal-oriented, whether the goal is to build something or simply to have good conversation. These ones are electrifying to be a member of. Shared endeavors and a sense of shared ownership seem to actually create energy.

Other communities, however, just depress everyone. A group of rules lawyers, whose shared energy is absorbed in the feeling that bad behavior is punished but good actions go unrewarded, is still a community. It’s just not a very pleasant one. One doesn’t go out and evangelize for such a community or for what it does. One doesn’t hope that others will come join it.

(There is a third kind of community to be mindful of, of course. A mob, like a depressive community, is a common failure mode of an energized community.)

The breakdown of trust is of course the most common reason that the first kind of community turns into the second. It’s easy, particularly as a leader or moderator, to feel betrayed by everyone when the crowd goes in a direction that you don’t want it to. And the fear of the mob is a powerful motivator. The temptation is to lock everything down, pretend that there is no community ethos but the one you provide.

But people don’t work that way. Clamp down on a community, and it turns sour; the community spirit becomes one of grumbling and nit-picking conformance to the stated rules. Spontaneous action for the common good, being unrewarded, goes away.

I’ve seen online communities go completely sour at this point, as the members in their turn feel betrayed by the moderators. Subsequent events just confirm the mutual hostility. Eventually many of these things break up completely.

This isn’t universal; sometimes the shared endeavor of the community is motivating enough to overcome the mutual mistrust. Gradually, a new balance is found; member behavior builds moderator trust and moderator trust reduces member resentment.

Communities may recover in time, but it’s not a pleasant process.

This rather discouraged rant has been brought to you by the letter M and the number 2.

Ink, turpentine, paper, water

For at least 1500 years, Japanese artists have practiced suminagashi, the art of marbling paper with ink floating on water. The marbler uses brushes to place alternating drops of black calligraphy ink and turpentine on the surface of a full basin, then lays a sheet of paper down to capture the resulting patterns. They look like clouds, or smoke, or the grain of twisted trees. Each pattern is unique, unlike in Western marbling, where the creator can reproduce essentially the same design many times.

Ink, turpentine, water, paper. It seems so simple.

And it is very simple, but only after you accept one thing: you are not in control of the outcome. The ink goes where it wills, and the marbler can only follow. There are tricks to give the pattern an overall direction, such as controlling the amount of ink and turpentine or gently blowing over the surface of the water. But the heart of suminagashi is trusting what you can’t predict or control.

I recently read George Oates’s essay about the ways that Flickr created its community: Community: From Little Things, Big Things Grow on A List Apart. Two particular paragraphs really jumped out at me:

Embrace the idea that people will warp and stretch your site in ways you can’t predict—they’ll surprise you with their creativity and make something wonderful with what you provide.

There’s no way to design all things for all people. When you’re dealing with The Masses, it’s best to try to facilitate behavior, rather than to predict it. Design, in this context, becomes more about showing what’s *possible* than showing what’s *there*.

Flickr’s history has proven her right. There are any number of wildly varying communities on the site, many of them either accidentally or deliberately experimental. Flickr groups are even cited as a case study in Here Comes Everybody, Clay Shirkey’s recent book on online community dynamics.

And now it’s our turn.

Last year, my company (MediaLab, which makes a library search software package called Aqua Browser Library) released our new social library software: My Discoveries.

The essence of My Discoveries is this: allow users to add information to the library catalog. Let them tag things, make lists of related items, fill in ratings, write reviews. Then let others see what they’ve done. Turn the patron’s interaction with the library’s catalog into a conversation with the catalog, and with each other.

I’ve been involved in both the design and testing. One of the core principles we’ve kept in mind throughout the process is that we cannot predict what people will do with it1. Designing and testing in the light of that kind of uncertainty is very different, and much more interesting, than working to a known, restricted usage profile. It affects everything we do, from what characters are allowed in list names to which statistics we want to gather. How does one design metrics to detect the unpredictable?

Tags, lists, ratings, reviews. It seems so simple.

  1. Of course, we are not so naive as to think that all the new ideas that people come up with for My Discoveries will be good ones. I moderate a web community in my spare time, so I know how bad things can get. As a result, I have put a lot of attention into the administrative interface—and I expect do more on it in the future. If we give users room to innovate, we have to give librarians the wherewithal to detect and clean up misbehavior.

The very unhappy path to Terminal 5

I see that the new terminal at London’s Heathrow airport is in the midst of another weekend’s disruption. Problems on the terminal’s opening weekend resulted in over 200 flights cancelled and a backlog of 28,000 bags. The chaos has already cost British Airways, the sole user of the terminal, £16m, and some estimates put the eventual cost around £50m.

Initial problems reported included the failure of either passengers or staff to find the car parks, slow security clearance for staff, consequent delayed opening of check-in desks, and multiple unspecified failures of the baggage handling systems. Once the initial failures occurred, a cascade of problems followed as passengers began to clog up the people-processing mechanisms of the terminal.

This weekend’s disruption has been blamed on “a new glitch” in the baggage handling system. I suspect that means that when they solved one set of problems they unmasked another. A spokeswoman assures us that they’re merely planning how to put an identified solution in place. Her statement doesn’t include any reference to the fact that these problems often nest, like Russian dolls, and that the new solution may uncover—or introduce—new problems.

Of course, my reaction was, “Did they test the terminal before opening it?” The errors shown include both functional errors (people can’t find the car park) and non-functional ones (the baggage system failed under load). No system is implemented bug-free, but the breadth of error type got me wondering.

Fortunately, the Beeb covered some of the testing performed before the terminal opened. Apparently, operation of the terminal was tested over a six month period, using 15,000 people. The testing started with small groups of 30 – 100 people walking through specific parts of the passenger experience. Later, larger groups simulated more complex situations. The maximum test group used was 2,250. BAA said these people would “try out the facilities as if they were operating live.”

Do 2,250 people count as a live test? Are they numerous enough to cause the sorts of problems you’re looking for in a volume test?

I plucked a few numbers off the web and passed them through a spreadsheet. T5 was designed to handle 30 million passengers per year, which comes out to an average of 82,000 per day, or 5,000-odd per hour in the 16-hour operating day (Heathrow has nighttime flight restrictions). These are wildly low numbers, because airports have to handle substantial peaks and troughs. Say that on the busiest day you get 150% of flat average, or 7,500 people per hour. Assuming 75% of the people in the terminal are either arriving from or heading toward London, and the rest are stopping over for an average of 2 hours, that’s about 9,375 passengers in the terminal at a given time.

9,375 is more than 2,250. You can,however, magnify a small sample to simulate a large one (for instance, by shutting off 2/3 the terminal to compact them into a smaller space). It’s not just a numbers game, but a question of how you use your resources.

Most of the testing documentation will of course be confidential. But I found an account of one of the big tests. I would expect that any such report was authorised by BAA, and would therefore be unrealistically rosy; they want passengers to look forward to using the new terminal. But still, the summary shocked me.

In fact the whole experience is probably a bit like the heyday of glamorous air travel – no queues, no borders and no hassle.

Any tester can translate that one. It means:

We didn’t test the queuing mechanisms, border controls, or the way the systems deal with hassled passengers.

In software terms, there is something known as the happy path, which is what happens when all goes well. The happy path is nice to code, nice to test, nice to show to management. It is, however, not the only path through the system, and all the wretched, miserable and thorn-strewn paths must also be checked. This is particularly important in any scenario where problems are prone to snowballing. (Airport problems, of course, snowball beautifully.)

Based on the account I read, these testers were set up to walk the happy path. They were not paid for their labours, but were instead fed and rewarded with gifts. I’m sure food and goodie bags were cheaper than actual pay, but they dilute the honesty of the exchange. We’re animals at heart, and we don’t bite the hand that feeds us. We like people who give us presents. Getting those people—mostly British people—to act like awkward customers, simulate jet lag or disorientation, or even report problems must have been like getting water to flow uphill.

Furthermore, look at the profile of testers mentioned: an ordinary reporter and a bunch of scouts and guides. I wish I believed that the disabled, the families with cranky children, and the non-English speakers were just at another table at breakfast. But I don’t. I suspect the test population was either self-selecting, or chosen to be easy to deal with. In either case, it didn’t sound very realistic.

It’s possible that there was another test day for people who walked the unhappy path, and that it wasn’t reported. It’s possible that they did clever things, like salt the crowd with paid actors to clog up the works and make trouble, and that our reporter simply missed those incidents.

But I’ve worked on big projects for big companies, and that’s not what I’m betting. I suspect there were very good test plans, but that for reasons of cost and timing they were deemed impractical. So compromises were sought in large meetings with mediocre biscuits. Gantt charts were redrawn late at night using vague estimates that were then taken as hard facts. Tempers were lost, pecking orders maintained. People assured each other that it would be all right on the night.

It wasn’t.

I wish I believed that the next time someone does something like this, they’ll learn the lessons from the T5 disaster. But that’s happy path thinking, and I’m a tester. I know better.

Wednesday Nights Are Update Nights

Probably because I keep them free due to the need to pack for the weekend at home, I seem to be falling into a pattern that includes blogging on a Wednesday. So how has it been, this last week?

Things really divide into separate timeframes, based on the two cities I’m living in at the moment.


Returning home on Thursday was exhausting. It was the end of a draining week, and I had been with my colleagues at the drinks before a company dinner (I had to leave afterward to get my flight). It meant I got to meet company founder Thijs Chanowski, known to most of my Dutch contemporaries as the producer of the children’s show De Fabeltjeskrant (British readers: it would be like meeting Oliver Postgate. American readers: think of one of the early founders of The Children’s Television Workshop). He’s a perfectly delightful gentleman, with a gift for telling stories, and I was sorry not to be able to stay for the meal (though I wouldn’t have understood the speeches anyway).


Friday was a very pleasant day with the kids. Both had the charm going full blast, and we did a lot of playing while the washing machine repairman came and replaced a couple of parts. Then we went to the Gilmerton crossroads to pick up Alex’s friend Murray, and I had a funny moment. We were going into the small supermarket on the corner, and I caught myself mustering my Dutch to deal with the transaction before I remembered that here, I speak the language! It was almost a disappointment, like a challenge balked at.

Martin and I did a lot of packing and arranging on the weekend, and even managed a bit of garden work. I’d like to get the back garden weeded and mulched before we go, because otherwise the dock and the dandelions will eat the place alive.

I got the chance to admire the gap in Alex’s teeth, and to have a number of very pleasant conversations with both kids. There was some cuddling, too. And a bit of grunching, toward the end of the weekend, because they are human, and they miss me.


Coming into Schiphol, taking the train to Amsterdam Centraal, and taking the tram to the flat in the Oud West was almost routine. It was certainly easy – Dutch public transport is well thought out and pleasant to use. And the flat I’m borrowing, which seemed strange and foreign when I first moved in, seemed much more homelike.

Monday morning, I started a different commute. I work north of the river Ij, which used to be the northern border of the city, but has now been surrounded on both banks. There are two ferries that go to the appropriate section of the city, one from Centraal station, one from a less well-known area. And the knowledgeable at the office had pointed out that if I could bike, I could take the lesser known ferry, which would be faster and more fun than the tram.

My landlord was willing to lend me his bike (on the condition that I lock it well – Amsterdam is bike theft central!). It’s a classic “omafiets” – a black banger of a bike, with no gears and coaster brakes. These bikes are ubiquitous in the Netherlands, primarily because they are virtually indestructible. They also weigh a ton and are not very fast unless you pedal like a maniac (like the colleague I commute with on occasion.)

So I’ve been commuting by bike. And it’s been wonderful, even on rainy mornings. How can you beat riding along a canal on an omafiets?

The only thing that takes some getting used to about this method of commuting is the other cyclists. They scare me. In the Netherlands, if a car hits a cyclist, no matter what, it’s the driver’s fault. And in Amsterdam, the cyclists know this, and ride accordingly. Red lights are really for other people. It’s unsporting to indicate where you’re going next – just veer over and let the other traffic figure it out after the fact. The only thing a cyclist will get out of the way for is a tram.

But it’s kind of fun, once you accept that the fiets conveys immortality. It also seems to grant exceptions to any consideration of practicality – I have seen a woman cycling in three inch spike heels and a tight miniskirt*. I have seen children in wee baskets on the fronts of their parents’ cycles. I have seen kids with bikes and training wheels being pushed along by an accompanying parent’s hand on their backs. I have seen a window cleaner who used a bike to transport the tools of his trade, including the ladder (carried in one hand, parallel to but longer than the bike). I have seen a man riding slowly while his dog trotted along beside him on a leash. And that was just this morning.

And work has improved as well. I got further into the system this week, and got the chance to do some testing (you know, what they hired me for). I’ve even found an interesting bug or two, though I’m not sure the guys looking at them are as pleased as all that. And I feel more at home around the office, less concerned that I’m going to violate some invisible norm or offend people unwittingly (now I violate visible norms and offend people on purpose. But I am a tester.)

One high point this week was dinner with Dave and Liz, the couple who let me use their flat the first weekend in Amsterdam, and hooked me up with the place I’m in now. Every conversation with them these days is really a set of markers for much longer conversations we want to have over time. It’s really something to look forward to.

And the other high point was that Martin flew over today and we signed the lease for the house. Both of us had been worried that something would fall through…the product of the previous experience is a slight nervous twitch. But the paper is signed and things are committed. With luck, we can move on to the other challenges: getting school and childcare places for the kids, getting the move done, changing a lot of addresses and defaults.

All in all, though, we have been lucky. Nice house, nice jobs, and enough resources to see us through the unexpected. I feel much more optimistic than I did this time last week.

(So you can all stop worrying now.)

* I’ve cycled in a skirt one day this week, but it was long and loose. Dutch bikes tend to enclose the whole chain, so things don’t catch in the gears.

3 Days of New Job

I’m afraid I’ve been busy with social things these past few days, which is why I haven’t blogged about starting my job. Well, it’s one reason, anyway.

It’s very funny how I can move into a city where I know so few people (Dave and Liz, and people I met interviewing at MediaLab, and absolutely no one else), and still find myself dining socially two nights in a row. Monday evening was with Dave and Liz, and was a very pleasant chance to get re-acquainted after years of intermittent contact. It was less of a conversation we had than a series of pointers to future conversations – I don’t think we actually finished discussing any topics at all.

Last night was much, much stranger. You see, when Dave and Liz came home, I moved out of their flat (the cat prefers them and she owns the place, really). I’m now staying at the flat of one of Dave’s friends, Patrice, whom I have yet to meet. And Patrice has friends who needed a place to sleep last night, en route to Schiphol. He’d offered them the flat, and there are beds enough for all, so I threw together something easy for dinner*, and had dinner guests from two degrees of separation. We had a delightful time.

But now that’s all done with, and I get an evening alone. There’s some sorting through things to do, since I fly home tomorrow evening. (Which is a good thing. I miss my bunnies.) But really, I haven’t any excuse to avoid blogging about starting work and how it’s going.

Well apart from one. I don’t know how I’m doing. I can’t tell. The learning curve is very steep – it’s a lot of information to take in at once. But more than that, I’m not a standard new joiner. They can’t just sit me down in front of an IDE† and tell me to go code. I’m the first tester, and the first QA person, and it’s a little unclear what they expect of me. I know what I want to do – some of it – but I don’t know how to do a fair bit of that (in terms of what to type into the keyboard to get things to happen, not what I want to have happen), and whether what I want to do will make the company happy.

I just don’t know. And not knowing, I’m prone to thinking the worst.

On the other hand, I had been sure I’d failed my driving test.


* Chicken breasts wrapped in Serrano ham, in a passatta and basil sauce, served with pasta and salad. Which sounds like an awful lot more work than it was, particularly when the guests then cleaned up the kitchen!

† Interactive Development Environment, the place where coders write their stuff

In Amsterdam

I’m in Amsterdam.

Specifically, I’m sitting in the flat I’m borrowing (and cat-feeding) from a couple of friends for the weekend. (They’re off in Brussels so Dave can turn 40.) It’s a nice place, right in the centre of the city. I’ve made the acquaintance of their very pleasant neighbour, and their cat Magic (black, naturally). Though Magic is shy, I think we’re coming to an understanding. Feeding her helped.

I’ve also made the acquaintance of their wireless LAN, remarkably easily. Whatever one can say about modern technology, I love the ability to take my computer with me and connect to my virtual world in minutes. I spend so much of my mental time in cyberspace, and that is going to make this move dramatically easier, because wherever I go, there it is. It’ll still be there Monday, when I move into the flat I have for the bulk of the month.

I’m going to go out and explore the area this afternoon. I want to get some food for the flat (though all the necessities are here), and generally settle into Amsterdam life. It’s going to be an interesting month, living alone here (weekends aside). I suspect, though that it will be largely a wasted opportunity, from the point of view of cultural exploration…I simply do not feel like walking all over the city and having adventures every evening, particularly when I’m doing something challenging during the days (travel and starting a new job both count in this context).

One thing I do find: coming to Amsterdam one day a week, as I did for about three weeks running, has given me a lot of comfort in making this move. The entire transaction up to this point has felt like business as usual. And I’ve done enough aimless wandering around the centre of Amsterdam itself to feel OK about the next few hours. I suspect that the discomforts will come in the form of little surprises, unexpected moments. But that’s better than being overwhelmed by all the changes at once.

Standing poised…

…at the edge of the high diving board, toes curled just over the edge, arms extended to their fullest length. I can feel the thrum of the board as I flex my calves a little, just scraping the soles of my feet on the rough surface. This is the moment before the moment, before I bounce down, then up, before I soar and slowly draw my hands together above my head, before the long inevitable plunge to the water.

Tomorrow morning I fly out to Amsterdam. Monday I start my new job. Although I will be back in Edinburgh every weekend in July, tomorrow’s trip marks the first stage of moving out of Scotland.

I take a deep breath…

Pieces falling into place

It’s been some time since I’ve blogged. Plans have been up in the air, and sometimes I can’t bring myself to write about things that aren’t yet complete. As Martin wrote in his Going Dutch entry, we are moving to the Netherlands this summer.

This is, naturally, terrifying. It’s been particularly scary for me to contemplate, because I had to find two very important things.

A job
Moving country meant moving work, and that’s a frightening thing. I joined the Royal Bank in 1997 – October would have marked ten years there, and I was thoroughly institutionalised after all that time. It was intimidating to even contemplate finding something else.
A house
Admittedly, unlike the job thing, househunting is for the benefit of entire family, and in theory I could fob some of the weight off on Martin. But I get emotional about my living situation, so it felt like it was really my worry.

So how has it gone, in the month and a half since I quit the Bank and started these searches?


Martin pointed a job ad out to me in late March, before I was even officially out of work. It was for a small company, MediaLab, which makes search software mostly used in libraries. It’s a tiny company, and a deeply cool one, writing interesting software and having fun doing it.

At his urging, I sent them a CV. When we got back from California, I had a phone interview, and made a strong connection with the people I talked to. They invited me over for a second interview in person in their offices in Amsterdam.

That went even better. I enjoyed the conversations and liked the people, and it was mutual. More importantly, from a business perspective, it was clear that my area of expertise and my approach to work will fill a need in their company.

So I got the job.

I start at the beginning of July, which sometimes seems a long way off. I find myself thinking about the work, and about sitting in that bright and friendly office while I do it. It’s been a long time since I looked forward to work.


We wanted to rent a house for a year, to give us a chance to try out the Dutch lifestyle before committing a lot of capital to it. But there aren’t a lot of spacious, affordable houses in commute radius of Amsterdam.

It’s also difficult to search for houses at a remove. (My friend who just moved to New Zealand can testify to this.) After poring over hundreds of advertisements on the internet, we finally identified one that looked nice, in a promising town. So Martin and I went across one rainy Monday to look at it.

It was awful. Cramped, grimy and grim, in the shabbiest neighbourhood. It was also not available for a year’s rental; the owners wanted to keep the option open to sell it (an endeavour in which I wish them luck). We straggled home after a discouraging day, ready to abandon the whole damned effort.

I tried to take a fresh tack on the matter the next few days, looking again at places we had eliminated, sending out emails to emails to estate agents. Then the phone rang.

It was an estate agent, calling based on a profile Martin logged on their website. He had a four-bedroom property, he said, just coming on the market for a year’s rental. In Wormerveer, a town in commute distance from my office. Large workroom as well, was I interested? I made interested noises, and he sent me pictures.

Then I was really interested. It’s a light, spacious place, converted from a schoolhouse. The owner, a painter, is taking his family to the Canary Islands for a year. I flew over on Tuesday to view it and the neighbourhood.

It was fantastic. The town charmed me, and the location of the house was particularly good (it’s right near the market plaza, two schools, shops, and some pleasant areas to walk through.) And the house itself was better than the photos conveyed, with an essential unity of light and design.

It didn’t hurt that I got on very well with the owner, the painter, who showed me round. We talked about aesthetics and the philosophy of art, bookbinding and lithography, history and philosophy (boring the estate agent senseless until he recalled another appointment). Practical matters will go easier with this channel of communication, but more importantly, I’m looking forward to future conversations.

So now I have a job and a house, and frankly, they’re both fantastic. What a good set of prospects to take into a challenging year!

OAT Completion Report: Secret Sonnet

A lesson to be learned from OAT
is that the planning which assumes a test
will only run just once requires the best
environmental outcome, that there’ll be
no faults to find, and that the personnel
will be available to run as planned.
This doesn’t happen – often tests are canned,
the system breaks, or scripts aren’t running well.
Each test should be assumed to run at least
two times, with some days left aside to do
investigations, and to test the new
code fixes some before they are released.
It’s no good planning that we’ll hit a date
if known retesting means that we’ll be late.

Originally written for work, in a continuous paragraph rather than broken into lines.