Why iPad Pilots are Like Device Reviews
Have you ever considered how well reviewers really get to know the devices in their hands? It’s definitely something you should think about, because it should affect how much weight you give to review.
There’s a post over on The Chronicle that I almost missed last week. It covered one professor’s experience with running a small iPad pilot program one semester. His article doesn’t appear to have any connection to reviews, but one point stood out in my mind.
The pilot included about a dozen grad students in the journalism program at the University of Kansas. They were loaned iPads that had been bought by KU just for the pilot seminar. All were in their 20s, and all were digital natives.The loaner iPads were loaded with about 50 apps, and students could add or remove any app they wished.
The opinions expressed by the students were mixed, but there’s one in particular that is worth noting. Students didn’t get invested in a device because they didn’t own it.
As the semester progressed, some students used their iPads less and less, ignoring them for a week or two at a time. One explanation for that is that students had no sense of ownership in their tablets. They knew they had to return them at the end of the semester. The Duke students that Davidson wrote about were able to keep the iPod Touches they received. Other universities have done the same with the iPad. That provided an incentive to learn the technology and find apps that would serve them in the long term.
My students said they saw little or no return on investing large amounts of time to personalize the iPad’s settings, organize the many apps, transfer their address books or even learn new apps that might make the iPad more functional. Why become attached to something they had to give back? students said.
I want you to consider how that might affect blogs and review units.When a blog is given a device, they won’t have the same interest in it as if they went out and bought it. Loaners simply cannot get the same level of investment, and the students have shown us that.
I buy most everything I have reviewed, and now I can see that it was a good idea. When I put my money into a device, I’m investing in it. Since I want to get the most out of my funds, I am moved to learn more about the device. I try to use it as I would any of my gadgets, and that helps me get closer to a real user experience.
On related note, I think that most any review done after less than a week of intense use might not have the depth it should. Consider this: when one of the major tech blogs posts review within days of getting a gadget, how well do they really get to know the device?
But let’s take this one step further. The reviews that I feel are the most reliable are the reports that are made after owning a device for a month or more. These are often anecdotal, and lot come from average owners. But they can contain a critical point that is only noticed by an experienced user. These are the people who should be listened to, not the major tech blogs.
Now, I know that I’ve just discredited some of my own reviews, but I can live with that. The important thing here is that gadget buyers get the chance to make an informed decision. (Plus I have reviews that no one else has, so I’m relatively safe.) And I don’t know that I’ve ever read a meta-discussion of how to read reviews, so I thought this was useful.
fjtorres January 18, 2012 um 8:48 pm
Puts a bit of extra credibility meat on the Amazon reviews, no?
Myself January 19, 2012 um 5:12 am
The Verge has adopted the tactic of posting the review, and *later* running the battery tests to determine, well, battery life. It really makes you wonder how long they’ve had the device in their hands. The last time I checked, for example, the Transformer Prime battery life had changed several times from 6 to 9 hours and somewhere in the middle – the reviewer seemed a bit confused. Man, it’s a portable thing: battery life is absolutely key and you had better be sure how long it runs.
"Uh, oh, you should make it through they day – but it’s a modern smartphone so keep your charger in the bedside table. Battery life will go down quicker if you sync Twitter and play games or video a lot."
D’oh! That’s the standard comment I have read in 80% of the phone reviews I have ever read. But it’s so obvious: everybody consciously adjusts his own usage of the phone so as to make it through the day while using it the most he can. And of course everybody charges his phone at night. Thanks for the info!
Then there’s the issue that bloggers are people with routines and established practices: if a radically new product comes along, it’s likely to be rejected unless it is from an established and/or cool brand. Which in 98% of the cases is American. Seriously, if the Padfone or the Note had been designed by a Silicon Valley company, every pundit and blogger would be salivating and praising their innovation, their boldness in trying something new, their outside-the-box thinking, yadda yadda yadda. As it turns out the Note has been on sale for two months in Europe but reviews in the US have been very superficial and many sites simple have not reviewed it. And that Lenovo TV at CES that was unlike any other TV you’d ever seen? Forgotten after 10 minutes.
Expanding a bit upon the last point, bloggers seem to assume that lands outside the US are uninhabited. The Xbox is the best-selling console (it isn’t or at least it’s very very close with PS3), everybody gets 2-year commitment with their phone (globally, most people are on prepaid or with no commitment in their contracts), everybody pays 100 dollars every month for cable TV with 3000 channels (WTF?), yadda yadda yadda.
And if any of the things above changes, such as Netflix challenging cable, it’s a mega-hyper-revolution that has disrupted the innovation landscape across the board and transformed the industry overnight. Although Youku and Tudou in China already do that thing, mind you. The use of "overnight" to describe decades-long processes such as the rise of online video is also a bit curious.
It even comes to the point of ignoring products that are not released in the US even though they’re obviously significant. And this also applies to business in general: billionaires outside the US simply "have" money, and in fact that’s a bad thing. They’re robber barons. Slim, for example, just happened to have friends in Government who helped him establish a massive monopoly….in many different countries and industries, it turns out. Not that he ever had a good idea or developed anything innovative or useful. Visionaries are not born outside the US, and monopolies are bad except when they are in the US.
Carlos Slim –> domination bad
Google/Apple –> domination good