It’s kind-of crazy to think that I work in a field that is not even as old as I am! There are conflicting accounts of what year we actually started to see web pages be actively optimized for search, but it was likely sometime between 1991 and 1997. For this post, I thought it would be interesting to discuss the history of SEO, and what I think the future of search will look like.
Even though SEO isn’t that old, there’s a lot to dissect. As different tech companies have had success and failures, there are varying levels of innovation and competition that have moved the industry forward. In this in-depth piece, I will discuss how the history of the Internet and Search evolved into SEO, and how competing search engines have impacted that history.
Ask Jeeves about the History of Search
Sometimes it’s hard to remember life before the Internet. (Especially if you’re a millennial, because the Internet pretty much started around the time we were born.) But when the Internet was new, there was all this information out there and no good way to find it. Pretty soon, different search engines started popping up, like AltaVista and Ask Jeeves. And Yahoo! We can’t forget Yahoo (unless you already have, which would be understandable). All these engines were trying different things to be successful. But most importantly they provided avenues for tracking the ROI of having a website.
And Then Came Google
In 1996, two Stanford Ph.D. students started experimenting with an algorithm for a research project they nicknamed BackRub. (Because they were developing a system that checked backlinks to determine the value of a site.) Those students were Larry Page and Sergey Brin, and their research project eventually became Google, which soon became and has remained the world’s leading search engine.
Search was a completely different thing prior to this. Google’s innovative use of backlinks allowed them to create a vastly improved search experience. The other guys were still mostly tracking how many times a website mentioned the user’s search query. It didn’t take long for people to figure out how to game the system by spamming text with keywords.
Google found a smarter way to rank pages that combines keywords and backlinks. They patented this technology, which was a very good idea. At this time, businesses started to recognize the value of having the top ranking on Google. This was when SEO was truly born. Companies started turning to their web developers and asking, “How can we be Number 1?” SEO strategy as a field started to emerge, as did less ethical strategies for improved search ranking. Whenever the Internet and money are involved, people will try to do shady shit. So people got creative and figured out how to buy links.
Continued Innovation
In the late 1990s/early 2000s, the Web became a lot bigger turned into millions and millions of pages. Google kept pace with this growth, and sought to improve its algorithms and fight back against people trying to game the system. They hired innovative software engineer Matt Cutts to head up the Google WebSpam Team. He was responsible for figuring out how people game the system, and working with engineers at Google to figure out how to prevent them from affecting search results.
Now the Google algorithm is fairly complicated, and uses hundreds of thousands (if not millions) of signals to make sure page ranking is still about the quality of content. Google has continued to improve its processes, and the search engine can now understand the context, not just the keywords people use. They have AI to understand what you are talking about, and they still view links as a heavy signal of a trustworthy site. They are also now using machine learning to figure out if there is a shady pattern happening in the background. So if you are artificially boosting backlinks, Google can find out. Google also now tracks and considers mentions of a brand or product. So if people are talking about you it adds value in your search strategy, even if they aren’t linking to your site.
The Impact of Search: A Case Study
If you want an example of how much SEO has impacted online commerce, just look at Amazon. In the early years of Amazon, they went all in on Google Adwords, so when you searched for something you wanted to buy, on the right you’d see an ad for that thing on Amazon. Eventually Amazon figured out that they didn’t need to pay a lot of money for those ads, when they could optimize for SEO and get the same results.
Amazon started with books and slowly added other products. Because of the convenience of shopping with Amazon, people clicked through to those other items. Those clicks gave Amazon a great boost in SEO. They are big for a reason. Every time they can offer you an additional product, they are effectively boosting their search ranking for almost every other product they sell. Google also ranks results based on number of clicks on SERP, and Amazon offers you thousands of things to click on. And the longer you stay on their site, the higher the rank they will get as Google sees session length as a positive indicator. Amazon is now the top e-commerce website, and is constantly expanding to provide new products and services. I don’t think that would be the case if they hadn’t been so strategic with their SEO strategy early on.
Search Engine Rivalry
Even though Google is clearly the leader in search, I think it’s still interesting to look at how the different search engines have influenced and even collaborated with each other over the years.
A lot of people don’t know this, but in 2000, Yahoo made a deal with Google to use their competitor’s technology to run Yahoo’s search engine. This deal came after Yahoo began to see they weren’t going to be able to compete with Google. Yahoo also differentiated itself by becoming more of a media site. If you click on these links right now you’ll see that Yahoo offers a huge range of content from its homepage, including news, entertainment weather and financial market updates. Google on the other hand is still a straightforward jumping off point for search. Although Google has a company has expanded into a suite of other products, most notably through the Chrome Browser and Gmail and through the acquisition of YouTube.
This arrangement between Yahoo and Google lasted a few years, until Yahoo dissolved it. After that, Yahoo struggled to keep up with Google’s technology. They struggled to keep up with Google in other ways, too. When Google launched Gmail, Yahoo had to up its email game and expanded free storage for Yahoo! Mail to 1GB.
In 2009, Microsoft decided to throw its hat into the search ring and launched Bing. As the newcomer, Bing initially presented itself as a visually stunning alternative to Google with a simple landing page backed by impactful photography. (And then, as you scroll down, you see a similar range of content to what you’d find on Yahoo.)
Google has remained in a very strong first-place among search engines. According to Net MarketShare stats from the last 18 months, Google has 78.63% of the search market share, Baidu (a Chinese tech company) has 12.72%, Bing has 4.60% with Yahoo! coming in fourth at 2.55%. Part of Google’s success, especially compared to newcomers, is that algorithms are iterative, which means they get stronger over time. That means companies diving into search 10+ years after Google are at an inherent disadvantage. It’s also hard for these companies to compete with Google’s ability to recruit and hire the top engineers.
The Evolution of Keywords
In order to improve search rankings, companies used to print out lists of keywords and command that their content teams insert those keywords into different parts of a website. These days, Google has become sophisticated enough that its search engines understand the context of your site. So it’s actually more valuable to write articles that are of real use to your audience. I think that writing around keywords sounds pretty unnatural and now Google agrees.
In my experience, clients can easily get hung up on the details of keyword rankings. Using tools like Google Search Console, you can see how many total keywords you are ranking for. A lot of companies will pick the 5,000 keywords they want to rank for, and focus on those.
But I would challenge that and ask if you are gaining new keywords. You would want to see your website grow proportionately in terms of traffic and keywords. If you are only tracking those 5,000 keywords, but you have 20,000 that you are ranking for, then you have tunnel vision. You have to be able to take a step back and look at the big picture.
You also have to consider the actual demand for the keywords you’re ranking for. You can have a rank of Number 1, but if nobody is searching for that keyword then there’s no point. Part of your SEO strategy should be verifying there is actual demand for the keywords you are targeting.
It does get pretty complex, and it’s amazing how far we’ve come from the first search engines that just counted keywords.
The Future of SEO
One of the big shifts we’ve seen in SEO is Google going towards more personalized search results. More and more, Google is giving users results that are specific to their interests and browsing history. For instance, if you are a film buff and you search for the word “red,” you may get a result about the Bruce Willis movie, RED. If you don’t have that interest, you might see a different result.
There’s no doubt that this makes things more difficult for SEO. A lot of optimization is based on how users in general behave and respond to content. We don’t necessarily have the capabilities to tailor our methods to individual users’ search patterns. More specifically, personalization makes it difficult to track KPIs and for us to understand if what we’re doing with keyword rankings is working.
I also suspect that search is going to get more sophisticated in terms of video. Google owns YouTube, so they have plenty of video data to work from. Google is scanning the audio of the video to see what is being talked about, transcribing that audio and running that through the algorithm. As this technology gets stronger and impacts search, it will drastically shift the way companies think about video. Right now it’s about having the right title, description and tagged keywords. Soon, the overall content of the video will be incorporated into search so content creators will have to think much more broadly about what they are posting online.
A lot of companies and brands just put something online without a lot of consideration for the SEO impact. As SEO evolves, I think content will get really complicated and a lot of companies will die if they can’t keep up.
I also think that Voice will impact SEO in a big way in the future, as it will drastically increase competition to be in the Number 1 spot. If you run a search for “What’s the best restaurant in London?” in a browser, you will get pages of options, and have the chance to browse the results. But when someone asks the same question of Alexa, Google Assistant or Siri, there will probably be just one answer. And everyone will want to be that top answer. Companies need to focus on video and audio to make sure they are ready for the next level of SEO.
And…
So there you have it, an overview of the history of search, as powered by Google and other would-be Leaders in Search. I think it’s important to remember that technology is always changing. When Google arrived, it drastically transformed how we use the Internet. You never know when things will shift, or some new innovation will impact how e-commerce is conducted.
What do you think about the history of search? Did I discuss anything here that surprised you? I’d love to hear your thoughts!