Nielsen’s Kindle reading speeds study was flawed

The results of a brief study into reading speeds were posted yesterday. The study compared the speed of reading with a:

  • paper book
  • PC screen
  • Kindle
  • iPad

You can find the report on the study here. I think this study was flawed. Something was measured, yes, but it was not the reading speed of the participants.

First, the author

If anyone besides Dr. Jakob Nielsen had done this study, I would simply point out the flaws and criticize the conclusions. But there is something you should know about Dr. Nielsen: he has _literally_ written the textbook on usability (several of them, in fact). I've used one in a class, and I still have it on my shelf because it  is a very good source of ideas. Have a look at his CV page and you'll see why I'm impressed.

Second, the study

The study had 24 participants, who each read 1 short story on each of the 4 devices (paperbook, PC, Kindle, iPad). The average reading time was 17 minutes.

I have 2 reasons why I think this study is flawed.

  • We don't know if any of the test group had used an ereader before. I'm going to assume they haven't. If the test group had used an ereader before then Dr. Nielsen would have remarked on it.
  • The reading time was too short. If the test group had used an ereader longer they probably  would have gotten used to them and read faster. This would have changed the results.

Third, the results

I'm just going to quote DR. Nelsen here:

The iPad measured at 6.2% lower reading speed than the printed book, whereas the Kindle measured at 10.7% slower than print. However, the difference between the two devices was not statistically significant because of the data's fairly high variability.

...

But we can say that tablets still haven't beaten the printed book: the difference between Kindle and the book was significant at the p<.01 level, and the difference between iPad and the book was marginally significant at p=.06.

So let's put the results in perspective: Among the people who (1) picked up an ereader for the first time and (2) briefly read on it, the reading speed was 6% to 10% slower than with a paper book. What he really tested here was not the reading speed. No, the data we have here is on adaptability. He answered the question: How much of a speed loss will you see when you pick up an ereader for the first time? That's the only question that can be answered from this data. The data says nothing about the reading speed of an experienced user, and thus it is at best incomplete.

Here's the thing: If he really wanted to test reading speed then he should have had a test group of people who already knew how to use ereaders. This way he could eliminate the chance that the test group is slowing themselves up with newbie mistakes.

Speaking as someone who has had his hands on goodness know how many ereaders: you cannot draw a valid factual conclusion about the usability of a device based on the first 20 minutes. (Emotional, perhaps, but that's a different matter.) The first 20 minutes is too heavily influenced by the user not being familiar with the device. I know this; I've been in that position more times than anyone.

P.S. There is a second set of results on user satisfaction. The PC rated low, and the iPad, Kindle, and paper book rated about the same. No big surprise there.

About Nate Hoffelder (11594 Articles)
Nate Hoffelder is the founder and editor of The Digital Reader:"I've been into reading ebooks since forever, but I only got my first ereader in July 2007. Everything quickly spiraled out of control from there. Before I started this blog in January 2010 I covered ebooks, ebook readers, and digital publishing for about 2 years as a part of MobileRead Forums. It's a great community, and being a member is a joy. But I thought I could make something out of how I covered the news for MobileRead, so I started this blog."

1 Trackbacks & Pingbacks

  1. Research: iPad and Kindle beat printed book in user satisfaction

Leave a comment

Your email address will not be published.


*