Skip to main content

Has Google Peaked?

There’s a natural cycle in the tech industry where old companies with stale ideas and technical debt (for example, investment in infrastructure). To give you an example from social media, Friendster was supplanted by Myspace was supplanted by Facebook. Also, awareness of this cycle is why Amazon maintains their "Day One" philosophy.

And now that cycle may be turning again in search engines. Twenty years ago Google beat out Yahoo and other early search engines by being better at scraping the entire internet and then ranking sites based on their linkage, but now it seems Google can no longer find everything.

Tim Bray writes:

I think Google has stopped in­dex­ing the old­er parts of the We­b. I think I can prove it. Google’s com­pe­ti­tion is do­ing bet­ter.

Ev­i­dence This isn’t just a proof, it’s a rock-n-roll proof. Back in 2006, I published a review of Lou Reed’s Rock n Roll Animal album. Back in 2008, Brent Sim­mons published That New Sound, about The Clash’s London Calling. Here’s a challenge: Can you find either of these with Google? Even if you read them first and can carefully conjure up exact-match strings, and then use the “site:” pre­fix? I can’t.

Why? Ob­vi­ous­ly, in­dex­ing the whole Web is crush­ing­ly ex­pen­sive, and get­ting more so ev­ery day. Things like 10+-year-old mu­sic re­views that are nev­er up­dat­ed, no longer ac­cept com­ments, are light­ly if at all linked-to out­side their own site, and rarely if ev­er visited… well, let’s face it, Google’s not go­ing to be sell­ing many ads next to search re­sults that turn them up. So from a busi­ness point of view, it’s hard to make a case for Google in­dex­ing everything, no mat­ter how old and how obscure.

My pain here is pure­ly per­son­al; I freely con­fess that I’d been us­ing Google’s glob­al in­fras­truc­ture as my own per­son­al search in­dex for my own per­son­al pub­li­ca­tion­s. But the pain is re­al; I fre­quent­ly mine my own his­to­ry to re-use, for ex­am­ple in con­struct­ing the cur­rent #SongOfTheDay se­ries.

Competition Bing can find it! Duck­Duck­Go can too! Both of them can find Brent’s London Calling piece, too.

Google is letting older and less-valued pages fall out of its index. That is not a huge issue (unless you need the content) but the fact that Google’s newer and smaller competitors don’t have the same issue is a sign that Google’s tech is getting old. Other companies have better ideas, and better tech.

If the cycle plays out the way it has in the past then Google has peaked, and is on the way out.

You will soon be switching to a competitor – DuckDuckGo, most likely. Bing may be backed by Microsoft but I have found it to be inversely useful (search for Dropbox, and it will return links to competitors).

image by betsyweber


Similar Articles


Fahirsch January 17, 2018 um 3:39 pm

I just searched for (without quotes) “Lou Reed’s Rock n Roll Animal album tim bray”. The result was in first place

Nate Hoffelder January 17, 2018 um 3:47 pm

Google fixed it after the post was published.

Wow January 17, 2018 um 3:47 pm

Wow, even if the claims in this article are true, that’s an insane leap, which isn’t even supported by the claims in this article.

The premise of this article is "Google doesn’t index old shit because they can’t monetize it (and are probably evil), so their technology must be lagging behind competitors who are desperate for anything that might get them a leg up against Google, even indexing old shit that no one cares about."

Really embarrassing, even for you, Nate.

fahirsch January 17, 2018 um 5:34 pm

Google does not need to index "old shit" now. It did it a long time ago. It doesn’t forget it either. But most of the time, "miraculously" seems to know what one is looking for: not coincidentally, other persons are also searching for the same thing

Mackay Bell January 17, 2018 um 8:11 pm

Exactly, the "innocent" explanation is that Google is outdated and trying to save money. I don’t believe that. I think the real problem is that Google has a huge financial incentive to push people who are looking for an an old article about Lou Reed to give up and just buy some Lou Reed on iTunes or Amazon Music or click to a Rolling Stone or NYTimes article in hopes of getting the same info. I’ve noticed Google becoming less and less useful for finding information and I’ve heard from many others who feel the same way. This is the classic problem with allowing a one company to have a monopoly on a critical resource.

Nate Hoffelder January 18, 2018 um 12:03 am

Now that is an interesting alternate explanation. You didn’t disagree so much as come up with an alternate explanation for why the cycle might be advancing.

Anna Erishkigal January 21, 2018 um 9:44 am

If you follow any prominent YouTube vloggers (especially on the alt-right), Mackay Bell’s theory fits in with what the vloggers are seeing with YouTube demonetization and their new policy which strips about 95% of "Partner Program: vlogger accounts of their ability to monetize completely.

It’s been a long trend, but this vlog by independent journalist Tim Poole (of former Vice fame / Occupy Wall Street) did a vlog a couple of days ago which might pique your interest?

VLOG "This is the end for 95%…"

anon January 20, 2018 um 7:49 am

What it likely is is
No one google was indexing is spidering had ever linked to tbray before
now that you (TDR) has linked to it, google has followed your link – and then will (if tbray is multi-cross-linked within itself – and doesnt have a robots.txt blocking such) spider-in all of tbray, causing tbray to then end up in google’s system

I’ve seen it myself with other sites too – strange weird little out of the way places, that for example a friend setup – no one knew about it and so it wasnt linked to from anywhere – I then added it to a netminding / change detection type service, to track it – and then next day it was in search engines!

fahirsch January 20, 2018 um 11:13 am

I believe most people don’t know that they have to tell Google and Bing that their site exists. And even better, have a sitemap that tell search sites all the pages the site has. In the case of WordPress there are plugins that automatically inform Google and Bing. When I write a new post in my site it takes at most a few hours to have it indexed.

Nate Hoffelder January 20, 2018 um 11:54 am

Except he had used Google to find that post before.

And this is a web guy, so i think he does know about sitemaps.

Write a Comment