Debunking the Claim that "the ‘Author Earnings Report’ is Misleading"
If you’ve caught up with today’s morning coffee post then you’ve probably already read Oscar Bernie’s criticism of the Author Earnings Report. That post was picked up by The Passive Voice blog yesterday and lead to a discussion which is now 292 comments long and counting.
While Bernie’s post was a good read, the ensuing debate illustrated a number of flaws with his arguments and I would like to address them.
To start, Bernie showed up in the comment section of TPV and argued his case. I have to give the fellow points for showing up at his own vivisection; that takes gumption, and it is why I am willing to look past his condescending remark about housewives, the ad hominem and false attack that TPV is "sponsored by Amazon", and the other ways Bernie shot himself in the foot.
In the Problems section of his post, Bernie raised a number of issues with the latest Author Earnings Report (he did not read the earlier reports, apparently). He then helpfully summed up his arguments in the comment section at The Passive Voice:
But give me a logical counterpoint to any of my assertions:
– Seasonal variation isn’t accounted for.
– Increase of indie books (change to sample size) makes the pie charts useless. If the market doubles but Big 5 stays the same, a pie chart appears to show a shrinkage that doesn’t exist.
– We don’t know the impact to the Big 5 without print sales.
– The overall market is growing and cannibalization doesn’t seem to be an issue.
I’ll discuss real things. But the number of logical fallacies here is staggering.
Let’s address his points one at a time, starting with seasonal variation.
Bernie objects to how the latest Author Earnings Report compares data from January 2015 and May 2015, and points to the problem of seasonal variation.
This is indeed a valid concern (it is for example why you read year-over-year comparisons in a company’s revenue reports). It is also why I went looking for the older reports when I covered the latest Author Earnings Report. Bernie did not bother to do so, but the data is there so this point is irrelevant.
What’s more, others have looked at the source data and concluded that seasonal variation doesn’t have an impact on relative market shares.
And while we’re on the topic of data, Bernie also demanded (in his post) that the AE Report provide more raw data. Apparently he skimmed the report and missed the fact that source data is provided. Bernie is welcome to go dig through it and crunch his own numbers.
Next he raises the issue of sample size.
Increase of indie books (change to sample size) makes the pie charts useless.
In a word: no.
If this were a valid complaint then it would mean that no consumer market survey is valid. Those surveys, whether they’re performed by Pew, BISG, Neilsen, or whoever, rarely use more than a few thousand participants to draw conclusions on millions. Consumer survey groups are never the same size, and they rarely have all the same participants from one survey period to the next, but they are always taken seriously.
The Author Earnings Report tries to solve the sample size problem by using a sample size as large as possible. To complain that it is not big enough is simply ridiculous.
Next, Bernie raises a point about print sales:
We don’t know the impact to the Big 5 without print sales.
That is irrelevant. This report has always been about digital and how much money there could be in the ebook market. Yes, digital affects print and print affects digital, but we’re talking about the digital market here. Print is irrelevant to this discussion of the size and value of the digital market.
Let me give you an example: this would be like arguing that a report on tablet shipments is irrelevant because it did not include laptop sales (because one market influences the other). Clearly the argument makes no sense.
Bernie then closes out his arguments with an unsupported claim:
The overall market is growing and cannibalization doesn’t seem to be an issue.
He offers no evidence to support this claim, so there’s no need for me to refute it.
All in all Bernie’s report was an interesting read, but as the many commenters showed over on The Passive Voice, it has serious flaws.
I can’t quote them at length here but if you have the time today, I would go read the comment section for this topic. It shows why I often feel the debate on that site is sometimes more valuable than the original source.