Posted on by Martin

Devices, devices, devices

Before I start on my own thing, let me first point you to what Brad Frost and Stephanie Rieger have already written on the subject of building a device library:

Without wanting to get into the debate over “sites” and “apps”, I think the distinction has some bearing on how you choose to spend your money in building a device library. Here’s how:

  • Site-like, or document-oriented web projects generally align with a 100% compatibility strategy: you want your content to render acceptably and be at least minimally usable on any web browser.
  • App-like web projects generally align with a partial compatibility strategy: your focus is on providing a set of features to browsers that meet a set of minimum requirements. (Although if your requirement is simply “webkit”, I may have to come and hurt you.)

Other markets are different, but StatCounter’s mobile figures for Europe in February 2012 show that 73% of mobile browsing comes from iOS and Android devices. (Some Android and iOS users may be using Opera Mobile or Mini, but that percentage is small.) Throw in a smattering of BlackBerry 6 and 7 devices, some high-end Nokias, and a few Windows Phones, and you could take this to mean that about 80% of mobile web users are working with a touch-screen device running a relatively modern browser.

If you go with the partial compatibility strategy, your device library should cover this 80%. Testing a feature-heavy site on BlackBerry 5 or Nokia S60 browser is going to drive you mad, and isn’t going to serve your core customer base. Because you’re concentrating on the medium-to-high end of the market you’ll spend more money per device, but you can get away with relatively few of them. I’d suggest a handful of Android and iOS devices, a BlackBerry 7, and a Windows Phone Mango. Make sure that the devices you get cover a variety of screen resolutions (small smartphone, large smartphone, tablet).

If you’re aiming for 100% compatibility, you need to focus your attention on the other 20%, because that’s where you’re going to spending the vast majority of your time debugging and fixing crazy bugs. Buy as many cheap, old, low-spec, oddball devices as you can lay your hands on. Any time someone says “WTF – you’re browsing the web on that thing?” award yourself special bonus points. Low-spec is good because if your site works well on a slow device, it’s only going to do better on a faster one. Because the 20% represents the long tail, you will need more devices, but they will be cheaper ones, so you’ll probably end up spending just the same amount of money.

Building a device library is not a one-off thing. It’s not like a developer workstation that you can get away with upgrading every 2-3 years. New devices and new OS upgrades are being delivered at a rapid pace. If you can afford it, you should be looking to add new devices to your library at least every six months. Again, if you can afford it, don’t upgrade the OS on old devices. Over time, they will become your legacy devices.

Finally, I consider it important to actually use the devices. Having a spectrum of devices sitting on a shelf in the office while you walk around with an iPhone 4S in your pocket will not get you exposed to the strengths and weaknesses of the different platforms and form factors. For example, if all you use is a touchscreen, you won’t understand what it’s like to browse with just a keyboard. (Try unplugging the mouse from your desktop machine, and see how that works out.)

For reference, here’s the stack of devices I have on hand for testing:

  • Nokia N95. 240×320 screen, S60 browser. The first phone I used for doing mobile web work. Released in 2007; an impressive piece of kit at the time. Unfortunately it has lost the ability to connect to my wifi, so testing with it is a bit of a pain now.
  • Nokia C3-00. 320×240, keyboard, no touch screen. Uses the Nokia Ovi browser by default, but also has Opera Mini built in. Cute little phone.
  • Nokia C5-03. 360×640 resistive touch screen with haptic feedback – it buzzes when you press the screen hard enough for it to register a touch/tap. The screen is small, but high resolution, and looks great. I used to think that it was a bit crap for web use because it only had a numeric keypad, but that was before I figured out that the qwerty keypad only shows up when you hold the phone in landscape mode on its left side, not on its right. Way to go, Nokia. Also, the “Create WLAN connection in offline mode” prompt should have died 5 years ago.
  • Motorola FlipOut. Android 2.1, 320×240 touch screen, and a keyboard. The keyboard doesn’t slide out, it rotates out from behind a corner. When it’s closed, this looks more like a pager than a phone. It’s cute and small, and I really like it.
  • Samsung Galaxy Ace. Android 2.2, 320×480 touch screen. Relatively cheap and versatile, because it can be upgraded to run 2.3 as well. (Buy two while they’re still selling them with 2.2!).
  • Samsung Galaxy Y. Android 2.3, 240×320 touch screen that pretends it’s 320×480. Cheap. Nice-looking little phone, but the touch screen is dreadfully inaccurate.
  • Acer Iconia Tab A100. Android 3.2, 1024×600. Piece. Of. Crap. Poor screen, slow, a capacitative “home” button that makes an accidental touch target no matter which way you hold it, and the battery won’t hold a charge in standby mode for more than 48 hours. The only thing it has going for it is its price. (That, and the fact that it fills the Android 3.x hole in my line-up.)
  • iPhone 3G. iOS 4.2, 320×480. The first iPhone in our household; I inherited it from Abi when she upgraded to a 4 in 2010. Useful as a legacy iOS4 device.
  • iPod touch 4th generation. iOS 4.3, 320×480 (retina). Gorgeous little device. This is my stand-in for an iPhone 4, because it has the same A4 chip and retina display resolution (although the screen is slightly lower-quality than the actual iPhone 4). I’m currently trying to figure out if I want to upgrade it to iOS5 to match the distribution of hardware in the wild, or keep it on iOS4 as a legacy device. It’s a cost consideration.
  • iPad (original). iOS 5, 1024×768. Still great.
  • BlackBerry Curve 8900. BlackBerry OS 5, 480×360 screen, keyboard. BlackBerry 5 is the OS That Will Not Die. BB5 devices are still being sold in volume because they’re cheap, have great keyboards, and people like using them for texting and messaging.
  • LG Optimus 7. Windows Phone 7 Mango, 480×800 touch screen. This was my first Windows Phone, but I don’t think it’ll be my last. I like WP7 as an OS. I like the sleek design, and the live tiles on the home screen. As a phone, the Optimus 7 is just okay. It’s big, heavy, the screen brightness is harsh, and the camera is only so-so. But I still found myself carrying it around a lot as my day-to-day phone.
  • Nintendo 3DS. This is what I mean by “oddball”. It has two screens, one on top of the other. The top screen has an effective resolution of 400×240 (landscape) and the bottom one is 320×240 (landscape). When you load up a page, the page loads into the bottom screen, but you can scroll it upwards. Only half the page is touch-capable (the bottom screen). The browser is a custom webkit build, made by Netfront. It scores 125 on html5test.com. It’s cheaper than most of the other devices in my library, and it plays Zelda? Come on, how can you not love this? The big disappointment is that you can’t embed 3d images in a web page. The top screen has to switch into a separate 3d-mode to show 3d images and video.

And here’s what’s on my shopping list for the next round of upgrades:

  • An Android 2.3 device with a larger screen. Android 2.3 currently holds the largest installed base, and a lone 320×240 device just isn’t sufficiently representative 13 March 2012: target acquired. Motorola Defy+
  • The new iPad with retina display. These are going to sell by the ton, and people will be using them to browse the web all the time. Apart from wanting one for myself, I can easily see the new iPad on its own accounting for more than 10% of all mobile browsing by the end of the year. 23 March 2012: target acquired. This thing is awesome.
  • A BlackBerry 7. A touchscreen-only or keyboard-only device would be cheaper; a touch+keyboard device would cover both categories, but it much more expensive. Also, not a huge market share. Not sure about this.
  • An Android 4. There aren’t enough Android 4 devices on the market yet to give it a significant amount of market share, and therefore to worry about it as a target for testing with. I’m trusting that it’s going to be mostly like Android 3.x, but faster. I might check out the Samsung Galaxy Tab 2 7″ when it appears. Otherwise, I’ll just wait for cheaper devices.

Further reading

CategoriesUncategorized

2 Replies to “Devices, devices, devices”

  1. Alternatively, hire a professional QA team such as provided by spriteCloud. We already have the most used devices, and constantly upgrade our device library to meet client needs.

    The basic principle still applies, however: you cannot cover all devices with testing, or the costs skyrocket. You’ll want to test on the most common subset of devices, and maybe even restrict some tests to only a subset of the subset. Either way, our test managers can help you create the list of target devices.

    @Martin: I hope you can forgive this shameless plug on your blog 😉 It just happened to fit well.

  2. No problem 🙂

    In the longer term, I think (and hope) that a lot of this problem will go away. Desktop browsers are now at the point where I can build a site using just Firefox or Chrome as my primary browser, and not feel too worried that it will look and perform like crap in anything other than IE6 and 7. Testing is still a different matter, but browser incompatibility isn’t nearly as much of an issue during development as it used to be.

    Mobile devices (desknots) are always going to have different form factors – that’s a basic fact that will remain, and we will always have to plan for different page sizes. But I think that in a few years we will get to a point where the vast majority of mobile browsers are “sufficiently capable,” and that at least developing with just a couple of devices on hand will not automatically be a cause for concern.

Comments are closed.