Mobile testing: you can’t really win (but you need to try)


Brad Frost has shared some great advice about mobile device testing. He and other mobile design advocates rightly suggest that anyone designing for the web should test on at least a handful of devices. But the above chart shows how futile that goal really is.

These are the mobile device stats for a very large web site that I work on with hundreds of thousands of visits per month from people all over the world. Seen in aggregate, the numbers aren’t terribly surprising: iOS dominates, Android adds up to also be very dominant, BlackBerry has dropped out of the top dozen, etc.

But take a closer look at what it means to “test on Android”. The top Android device here, the HTC Evo 4G, registers at 5%. Then Android devices quickly plummet to just 2% and lower.

So when you want to “test on Android”, which 2-5% of your mobile traffic do you want to test if you’re only going to buy one Android device for testing?

And even adding up the remaining 10 of the top 12 devices after iPhone and iPad only yields 18.9%, still under iPhone and iPad on their own.

Does this make mobile testing too much to deal with? No, I certainly don’t disagree with Brad nor anyone else who advocates mobile device testing. It needs to be done, and any device testing is better than no device testing.

But we need to start acknowledging just how ridiculous this is, and how unsustainable it is. Because it keeps getting worse. Android devices are like bunnies: there are more of them every quarter, and they’re all different. The above chart looks totally different if I just change the dates by three months. Yes, it’s that insane.

It’s one thing for a design agency to try to keep up, if it’s a mobile-savvy agency whose leadership understands the value in buying boxes of test devices to be thorough in their testing. But this is an entirely different problem for a freelance or in-house designer. Because let’s face it, when you dip below 5% with a desktop browser, you typically can afford to ignore it (or at least not worry about it that much).

Now you not only have to pay attention, you have to spend money on a device to cover that tiny percentage. Guess how well that discussion is going to go with most people’s bosses, especially when it has to end with, “Oh, and by the way, this will all change in a few months and we’ll need to buy more.” Right.

Again, I don’t mean to suggest that mobile device testing is itself futile. It’s critically important. The trouble is, we need to stop acting heroic about it and keep suggesting that all good web designers have to try to keep up with the intense fragmentation that is mobile today. Sure, designers with the right budgets can try to keep up. But keeping up is a constant effort, not a periodic effort.

In the end, it’s not just inconvenient to test thoroughly on most mobile devices: given the fragmentation of Android and the rate of new devices being released, it’s actually impossible. And what this means for the majority of the web — the web that isn’t designed by agencies with mobile device budgets, and rather is designed by in-house staff and freelance designers — that’s a problem that has no convenient solution without additional standardization.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s