How Search Engines Work?
A search engine is just another type of computer software (more or less) that works to index web pages into a database based on a quick scan of each page’s content.
Think of it like speed reading for a specific topic – you quickly scan material after material, looking for specific words to jump out at you. This is like a search engine – only a search engine does digital speed reading… and, of course, is constantly evolving in its abilities.
However, search engines don’t just do it all on their own; they bring in their buddies to help, sending spiders out to crawl the web. Those spiders then consolidate their findings and present it to the search engine to rank and summarize your site, pages, and information – alongside all of the other applicable or relevant sites.
Search engines work off of complex algorithms that are constantly changing – this is why the rules of SEO also constantly change; to keep up. There isn’t necessarily a “how-to” manual that always fits the SEO bill, but there are some rules that have stayed consistent among the change, as well as new rules and tips that have emerged alongside the new algorithm.
Exclusive SEMrush Deal
There are currently over 1 million people using SEMrush for their website SEO and content marketing. Signup for a trial using our special link and you'll get extended 14 days trial period (credit card info required) > Click Here
What is Search Engine Optimization (SEO)?
Search Engine Optimization (SEO) is the process of fine-tuning a website to try and achieve higher rankings in search results. SEO is done partly based on:
- One’s understanding of how machines interpret the searchers’ intent and matches that to web content (the search algorithm) and,
- Estimations on how humans interact with content they see online.
SEO has become extremely complex. To date, there are more than 200 ranking factors (parameters that affect a web page’s ranking) agreed on and recognized by Internet marketers.
These factors include users dwell time, link anchor text, keywords in URL, content length, TF-IDF, title tag, meta description text, web page loading speed, keywords in image alt text, number of outgoing links, number of incoming links, LSI keywords, search result page click-thru-rate (SERP CTR), and so on.
These factors were recognized by the majority because they were either verified by a Google spokesperson or proven to be (at least somewhat) effective in experiments and case studies published by well-known SEO experts.
Many, myself included, believe the number of critical ranking factors are far greater than 200. Each of these factors carry a different weight in different search result pages – which makes SEO incredibly (again) complex and hard to be explained. Some have called SEO more of an art than a science.
I am not going to dig into the details of these 200+ ranking factors. My objective with this article is to give you a quick overview on how search engines work today and share a list of the most critical SEO items to look into.
As Google holds more than 90% of today’s search market volume, I will interchange the term “search engine” and Google loosely in my article.
How SEO Worked in 2005?
This is how I did SEO 20 years ago:
- Run over a set of keywords at Overture (now gone) or Google Adwords Keyword Tool to determine the search volume for each keyword.
- Select a set of 30 – 50 keywords based on search volume and market competition. Target search terms with higher search volume but lower market competition.
- Segment these keywords into 10 – 15 topics. Each topic should consist of one primary keyword and a few other secondary keywords.
- Produce content on the topics – make sure that primary keywords are in page title tag and linger secondary keywords in page headings (H1, H2, H3, etc).
- Include beautiful images and keyword-rich alt texts to each of them.
- Interlink important money pages site-wide from header and footer
- Send out as many emails as you can to other webmasters and ask them to link back to your webpage using your primary keywords as anchor text.
- Buy backlinks from other websites if you have extra budget.
- Repeat step 1 – 6 endlessly.
Page titles, keywords selections, links, anchor texts, content freshness… This was largely how I built multiple high-traffic affiliate websites and blogs in the 2000's.
While this method may still work in a limited sense today, it is no longer an effective approach. The landscape in search and web technology has changed so much – it’s simply impossible to achieve the same good results using this method.
Why? Because search engines and the Internet works differently today.
Today’s Search Engine is…
Highly Secretive
Searches today are mostly encrypted – this means we can no longer fully see what users type in their search bar to reach our website. The most accurate search data we can obtain today comes from a handful of SEO tool providers that purchase click-stream data from third party brokers.
And not to mention – usage of ad blockers and VPNs are also blocking how data is shared among smaller site owners. We can no longer accurately see how many searchers are coming to our site and where they are searching from.
Frequently Updated
In 2009 Moz counted 350 – 400 changes in Google search result pages. In 2018 – the count went up to 3,234. With the excessive funding and human resources, plus theoretically unlimited computing power, Google is tweaking and developing and improving their search engine in a much quicker pace today.
A Google Spoke person once famously said:
In 2019 we ran over 464,065 experiments, with trained external Search Raters and live tests resulting in more than 3620 improvements to Search.
For updates that are expected to have a big impact on websites though, Google will announces it publicly on Twitter here. For smaller and minor updates, however, you'll have to rely on words from SEO forums and SERP Trackers from major SEO tool sites like SEMrush (see screenshot below).
Personalised
Google now serves highly-personalized search results to individuals based on individual preferences and web browsing history. The device you were using, such as mobile phone brands, tablets, desktops, smart TVs, and so on also factor in.
Even your behavior is also analyzed and contributes to some extent. For example, your usage history like what sites you visited, videos liked or shared, apps you installed on your smart phones, and other interactions.
Then there is the way you interact with search results (websites you clicked through, things you searched previously, ads your encountered, etc). These combine to dictate the next results you get from your Google search. My top 10 search results will very likely be totally different than yours.
Also read – DuckDuckGo, the Search Engine that Doesn't Track Your Search Activities
Cross-Platform
Searches are performed on various types of devices – which often represent different intent to search engines. For example – searchers searching for “aglio olio” on desktop are more likely to be looking for a recipe; but searchers searching for the same thing on mobile may be looking for an Italian restaurant. Even if you have the exact figures in a keyword search volume, it will be difficult to estimate the volume of traffic you will get.
How to SEO Today?
The biggest challenge for today’s SEO practitioner lies in execution, not know-how.
I couldn’t agree more with Kevin Indig’s segmentation of modern SEO into two categories –
- Macro level, which involves technical aspects such as website architecture design, UX optimization, website internationalization, and so on;
- Micro level, which involves focused content and on-page optimization such as intent matching and content tweaking.
Thing is, you can no longer come up with a set of fixed procedures in SEO and apply it to all websites and pages equally.
Every industry is unique.
Every website is unique.
Every intent behind a search is unique.
SEO is no longer a standalone marketing “tactic”; but something to be incorporated into your web development and content production process. To rank high on Google and grow your website, you need a continuous-improvement action plan that looks at both macro and micro-level pictures.
In that action plan, here are the five areas in your website that you must improve and optimize continuously.
1. Create Relevant and Useful Content (Duh)
Do – Produce web pages (and your website) that serve a clear purpose to your users. Constantly update and add value to these web pages.Ultimately your website should provide useful content that users can’t find elsewhere on the Internet.
If you are new, a big part of your SEO efforts would be spent in content audit. Ask the following questions.
- Is your content up to date and presented clearly?
- Does your content carry enough depth (and value) to the users?
- Does your content cover enough relevant topics to the users?
- Does your content display Expertise, Authoritativeness, and Trustworthiness (E-A-T)?
A news page is only beneficial to the users when it’s reporting the most recent or important events. A shopping page should provide all necessary information about the product and make a strong case for a sale. A how-to tutorial should provide complete A-to-Z information – in the form of text, images, or videos – on getting a task done.
2. Link In and Link Out Wisely
Do – Link to your key web pages internally frequently (without jeopardizing your website user experience). Link out to other relevant and useful web pages on the Internet. Get other relevant websites and blogs to link to you.
Links on the Internet are like votes in the real world – except that different links carry a different weight in search rankings. A link from a highly trusted site, Nasa.com for instance, has more power than a link from a web directory that links out to 500 different websites from one page.
Your main objective in link building, is to obtain as many “good” links as possible.
Different SEO methods approach link building differently.
Some methods recommend producing good content that attracts links naturally (people tend to link to content they find useful or interesting); while others obtain links by trading – money (sponsorship and ads), good content (guest posts), business relationships (networking).
Each of these approaches may or may not work for you. The key in this is to figure out what’s your strong suit and select a few suitable link building tactics.
3. Write Appealing Titles
Do – Write keyword-rich titles that attract users to click through to your site from search result pages.
Your page title does two things in SEO:
- Assist search engines in understanding your web page content
- To help promote your web page on search result pages
A title tag is limited to 65 – 70 characters. Important keywords and key value propositions should come at the beginning of your sentence.
4. Match Searchers Intent
Do – Check SERP for your targeted keywords to understand what Google regards to be the intent of the search. Revamp your page with new formats and additional elements to match the search intent.
“Search intent” is the goal a user is trying to achieve when performing a search on the Internet.
Search engine segments search queries into three different intent classes (quoting Andrei Broder’s paper):
- Navigational The immediate intent is to reach a particular site.
- Informational The intent is to acquire some information assumed to be present on one or more web pages.
- Transactional The intent is to perform some web-mediated activity.
Traditionally, searchers are usually very literal (in most cases) and tend to search for exactly what they want. Hence the basic idea of SEO is to match your site content as closely to as many relevant keywords in each search as possible.
Modern day SEO requires a little more than that. Not just that you content need to match searcher’s queries, but how your content is presented also makes a difference in intent matching.
To understand what Google regards to be the intent of a search, look at the top ranking pages for your target keywords. Compare how are your web page different than theirs. Revamp your page with new formats and additional elements to better match the search intent. You can gauge effectiveness either via how many users click through to your site or have users that stay longer.
5. Improve User Experience (UX)
Do – Emphasize on UX when designing your web page. Run A/B test regularly to improve your website user experience.
To help keep users who come to your site more engaged, more than just the draw-in is required. Your readers are your customers and leaving them with a positive impression is invaluable. This means you need to offer them security, a smooth browsing experience, and a pleasurable stay.
A few basic examples…
Using an SSL certificate will not just help your users secure their data while connecting to your site but also lets search engines know that your site is safe for them to direct traffic to.
Users who have to wait for a web page to load often get impatient and leave, so make sure your site is also optimized for speed.
Lastly, even though ads and popups can be an effective way of driving revenue, these can become intrusive to the users’ browsing experience.
Conclusion: SEO is a Journey, Not Destination
There are many companies and individuals today that offer SEO services. Before engaging them, remember that SEO is a journey and not just a destination. As websites and content evolves, SEO requirements will change.
Search engines are constantly changing the way their algorithms work as well, which means that you will never have the ‘perfect SEO solution’. They key lies in understanding, experimentation, and dedication – a lifetime’s journey, so to speak.
Frequent Asked Questions on SEO
SEO stands for Search Engine Optimization.
As mentioned, SEO is the process of optimizing a website to achieve higher rankings in search results. SEO is done, partly based on one’s understanding of how search alrorithm works and partly based on estimating how humans interact with their search results.
A “Frequent Asked Questions” page is always useful from a user's perspective. Carefully planned and built FAQ page works well as a sales tool and increase the number of content on your website (number of words, etc) and hence, increase your chance to appear in relevant searches. FAQs, when marked up with structured data (this one for example), increase your chance to be featured in Rich Search Results and (theoretically) help draw more clicks to your website. Refer to Google's and Bing's guide for more details in website marking.
Backlink is hyperlink that links from a webpage to your website. A backlink, also known as inbound link, are of the important ranking factors at Google.
Yes and no. There are plenty of helpful SEO guide on the Internet – so it's not difficult to get started and do it yourself to save money. Bear in mind, however, that SEO is very time- and labour-consuming.
Absolutely. Based on my study on top 400 freelancer profiles at Upwork, SEO charges, on average, $23.68 per hour. The fee goes as high as $175 per hour. Personally, I feel it's reasonable to pay $1,000 – $2,500 per month for a good long term SEO service.
Start by reading this guide and observing what other website owners are doing with their websites. Use SEO tools like AHREFS, SEM Rush, or MOZ to find out what others are doing to improve their search rankings.
