There’s a natural cycle in the tech industry where old companies with stale ideas and technical debt (for example, investment in infrastructure). To give you an example from social media, Friendster was supplanted by Myspace was supplanted by Facebook. Also, awareness of this cycle is why Amazon maintains their “Day One” philosophy.
And now that cycle may be turning again in search engines. Twenty years ago Google beat out Yahoo and other early search engines by being better at scraping the entire internet and then ranking sites based on their linkage, but now it seems Google can no longer find everything.
Tim Bray writes:
I think Google has stopped indexing the older parts of the Web. I think I can prove it. Google’s competition is doing better.
Evidence This isn’t just a proof, it’s a rock-n-roll proof. Back in 2006, I published a review of Lou Reed’s Rock n Roll Animal album. Back in 2008, Brent Simmons published That New Sound, about The Clash’s London Calling. Here’s a challenge: Can you find either of these with Google? Even if you read them first and can carefully conjure up exact-match strings, and then use the “site:” prefix? I can’t.
Why? Obviously, indexing the whole Web is crushingly expensive, and getting more so every day. Things like 10+-year-old music reviews that are never updated, no longer accept comments, are lightly if at all linked-to outside their own site, and rarely if ever visited… well, let’s face it, Google’s not going to be selling many ads next to search results that turn them up. So from a business point of view, it’s hard to make a case for Google indexing everything, no matter how old and how obscure.
My pain here is purely personal; I freely confess that I’d been using Google’s global infrastructure as my own personal search index for my own personal publications. But the pain is real; I frequently mine my own history to re-use, for example in constructing the current #SongOfTheDay series.
Google is letting older and less-valued pages fall out of its index. That is not a huge issue (unless you need the content) but the fact that Google’s newer and smaller competitors don’t have the same issue is a sign that Google’s tech is getting old. Other companies have better ideas, and better tech.
If the cycle plays out the way it has in the past then Google has peaked, and is on the way out.
You will soon be switching to a competitor – DuckDuckGo, most likely. Bing may be backed by Microsoft but I have found it to be inversely useful (search for Dropbox, and it will return links to competitors).
image by betsyweber