SpamBayes

Shortly after giving up on the SAProxy anti-spam tool the other week, I found an alternative tool that is doing the job very nicely: SpamBayes. It’s an add-in for MS Outlook that acts as a Bayesian filter for your email. (Big, big thanks to Anders Jacobsen for pointing it out.)

Whereas SpamAssassin (and hence SAProxy) uses a fixed set of rules to weed out spam, SpamBayes uses statistical methods to analyse your incoming mail. (See the article “A Plan For Spam” by Paul Graham for an explanation of the technique.) When you first set it up, you show it a bunch of mail that is good (“ham”) and a bunch of mail that is known to be bad (“spam”). It uses this to construct a profiling database. Each new piece of mail is checked against this database to figure out how likely it is to be spam. If it fits the profile, SpamBayes will automatically toss it into a “Spam” folder for you.

The neatest thing, though, is that the system is constantly adapting to new spam as it comes in. If a piece of email comes in that is spam, but doesn’t trigger the filter, then you click a button marked “Delete as Spam.” The offending message is added to the profiling database, so reducing the likelihood of similar junk mails getting through in future. Likewise, if an email is falsely tagged as spam, or as “possible spam” if the system is unsure, you click on the button marked “Recover from Spam.” This tells the profiling system that it made a mistake, and that it should adjust its probability weightings again. The system learns. SpamAssassin doesn’t, and so you have to keep updating it as time goes by, and as spammers learn to circumvent its fixed rules.

So far, it has been pretty good at catching incoming spam, but not quite as successful as SpamAssassin was. That’s probably because I didn’t have many examples of spam lying around in my mail to help it build up its initial profiling database. If you plan to use SpamBayes, I suggest you hang on to some of your recent spam so you can use it for teaching purposes.

An interesting side-effect of using a spam filter on my inbox is that I find incoming spam much more interesting now. Both SpamAssassin and SpamBayes allow you to see what it was that caused them to flag a particular piece of email as spam. SpamAssassin shows you what spam rules were triggered by a junk message, and SpamBayes shows you what words in the email contributed most to its overall Spam probability rating.

Ironically, this geeky fun factor means that I read (some of) my spam a lot more closely than I ever would have before. But in a forensic kind of way. Don’t be thinking that I actually spend time considering whether I really need some more Viagra this month. I keep careful track of my own supply, thanks.

Erm…

Related entries:

Related Links

Half-Life 2

Holy crap… If you haven’t seen the in-game footage of Half-Life 2 yet, go and download it now. Screenshots have been appearring for the last month or so, and they’ve looked impressive, but not spectacular… But the video just blew me away. The lighting, terrain models and textures make it look like real life. The character models aren’t quite cinema quality, but they’re still shockingly good.

Wow. Looks like I’ve got until 30th September to upgrade my PC.

Back pain

I don’t know what I did to my back, but…owwww.

I’m typing this standing up, with my keyboard raised up on my desk by a couple of volumes of Durant’s “The Story Of Civilization”.

Metroid Prime

Yay! I just finished Metroid Prime earlier this evening. It’s an amazing game, and I’m pretty sure I’m going to have to add it to my all-time favourites list.

The visual design of the world is astonishing, the music is fabulously atmospheric, the puzzles are accessible without being overly simple, and the reward structure is brilliantly judged to keep you playing just that little bit more. The enemies throughout the game become increasingly tough, but the learning curve is tuned to help you get past them before they become frustrating. The bosses are bosses are tough–as they should be–but rarely do you get the impression that they’re impossible. It’s always a case of, “if I just try it one more time, I’m sure I can beat it!”

When I bought our Gamecube back in March, it was Mario Sunshine and Super Monkey Ball 2 that were the compelling reasons to buy it. Well, the Monkeys are up on eBay right now because they’re so damn frustrating, and Mario is coming dangerously close to that same fate. Just like with Super Mario 64, we’ve got to a point where it feels like a chore to have to get the next gold star/shine/silly-ass flying pig to progress to the next stage.

But Metroid Prime? That ought to have been the real reason I bought the ‘Cube. My thumbs still hurt, but it’s the pain of satisfying accomplishment.

Diet, end of week 4

I’m down below 71kg now, despite having been a bit naughty yesterday. It was my dad’s birthday, and we all went out for lunch at Grape in St. Andrew Square. I couldn’t resist their burgers.

On the other hand, I only ate about a third of the frites that came with it, and finishing the burger actually turned out to be a struggle. I’m just not used to eating so much food any more. It kept me full up for the rest of the day, too. In the evening I wasn’t hungry at all, and all I ended up eating was some pineapple and yoghurt.

We’re off to Rome on holiday in a week and a half. Although my long-term goal is to get down to 67kg, in the short term I’d like to be down at 70kg for Rome. (Because its’ extremely likely that we will not be doing diet eating while we’re there. Going to Italy and not eating lots of yummy Italian food? Yeah, right.)

Barenaked

If you’re a fan of the Barenaked Ladies, you’re missing out if you’re not reading their blog. The guys are in the studio recording their new album right now, and they’re posting little stories, anecdotes and ramblings about how that’s going on. They’re also uploading cool photos, video clips (have a look at “Ed Robertson, Father of Ragtime Piano”), and instructions on how to fold paper airplanes.

On a separate but related note, something finally makes sense to me: why the Barenaked Ladies had a guy in a chef’s outfit playing shakers on stage with them on the Maroon tour. It’s all to do with their video for “Pinch me”, which I hadn’t seen until today. I’ve just been watching the “Barelaked Nadies” DVD, which my parents brought back for me from their trip to Florida. (I had thought it wasn’t available here in the UK, but that’s because Amazon.co.uk has it misfiled.) It’s a collection of their music videos, and some live footage. It also features the band giving commentary on the videos. Very cool.