How search engines crawl websites
Think of search engines as tourists with only a road map to explore the area. The map is the only reliable source of information to locate attractions in the country. The internet functions as the map, and websites are like the attractions. Search engines send crawlers through every website online. Crawlers are small programs that identify and report web page components. They gather data about the websites and index the information under separate categories and keywords, creating a list of sites that meet the query’s criteria.

How websites are indexed
Search engines use their own proprietary algorithms to index sites. While some engines prefer sites with a lot of backlinks, others prefer older domains, and some prioritize social linking and related activities. Search engines regularly change their algorithms to keep up with searcher’s demands and use feedback to provide better results than their competitors.

TIP: It’s important to allow, or even help search engines easily crawl your website. It can mean the difference between higher and lower SERP (search engine results page) rankings. Site structure will be key and luckily, search engines like Google and Bing provide site owners a few options to help them crawl your website: sitemaps and a file called “robots.txt.”
A sitemap is a list of available pages crawlers love to use. They enable search engines to easily index pages that point back to a site so searchers can find them. Sitemaps make it easier to crawl websites and pages, so they need to be search engine-friendly. The second method to help search engines crawl your site is using a file Robots.txt., this document promotes stronger search engine optimization opportunities and lets site owners give instructions to search engine crawlers. They can direct how to process individual pages or even notify Google and Bing which pages should not be crawled (and indexed.)

Search engine algorithms
Search engines like Google, Bing, Yahoo, DuckDuckGo, etc., use different algorithms to help determine which sites are the best matches for a search query. Some measure a site’s popularity, the amount of time people spend on a site, or content quality and site authority. Each of these factors give search engines a full-spectrum view into how valuable your website content is and how relevant it is to a user’s search query. Google has a few, well-known algorithms with quirky names like Google Panda, Hummingbird, Pidgeon, Pirate, Payday, and more. Each algorithm serves a particular purpose. Some algorithms like Panda and Hummingbird, focus on content quality while others like PayDay concentrate on cleaning up search results for traditionally “spammy” searches (think, “payday loan,” “credit loan” and other spammed keywords.