Site Crawling

 

What is Search Engine Crawling?

Crawling is the process used by search engine web crawlers (bots or spiders) to visit and download a page and extract its links in order to discover additional pages.

Pages known to the search engine are crawled periodically to determine whether any changes have been made to the page’s content since the last time it was crawled. If a search engine detects changes to a page after crawling a page, it will update it’s index in response to these detected changes.
 

How Does Web Crawling Work?

Search engines use their own web crawlers to discover and access web pages.

All commercial search engine crawlers begin crawling a website by downloading its robots.txt file, which contains rules about what pages search engines should or should not crawl on the website. The robots.txt file may also contain information about sitemaps; this contains lists of URLs that the site wants a search engine crawler to crawl.

Search engine crawlers use a number of algorithms and rules to determine how frequently a page should be re-crawled and how many pages on a site should be indexed. For example, a page which changes a regular basis may be crawled more frequently than one that is rarely modified.
 

How Can Search Engine Crawlers be Identified?

The search engine bots crawling a website can be identified from the user agent string that they pass to the web server when requesting web pages.

Here are a few examples of user agent strings used by search engines:

  • Googlebot User Agent
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Bingbot User Agent
    Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  • Baidu User Agent
    Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  • Yandex User Agent
    Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Anyone can use the same user agent as those used by search engines. However, the IP address that made the request can also be used to confirm that it came from the search engine – a process called reverse DNS lookup.
 

Crawling images and other non-text files

Search engines will normally attempt to crawl and index every URL that they encounter.

However, if the URL is a non-text file type such as an image, video or audio file, search engines will typically not be able to read the content of the file other than the associated filename and metadata.

Although a search engine may only be able to extract a limited amount of information about non-text file types, they can still be indexed, rank in search results and receive traffic.

You can find a full list of file types that can be indexed by Google available here.
 

Crawling and Extracting Links From Pages

Crawlers discover new pages by re-crawling existing pages they already know about, then extracting the links to other pages to find new URLs. These new URLs are added to the crawl queue so that they can be downloaded at a later date.

Through this process of following links, search engines are able to discover every publicly-available webpage on the internet which is linked from at least one other page.
 

Sitemaps

Another way that search engines can discover new pages is by crawling sitemaps.

Sitemaps contain sets of URLs, and can be created by a website to provide search engines with a list of pages to be crawled. These can help search engines find content hidden deep within a website and can provide webmasters with the ability to better control and understand the areas of site indexing and frequency.
 

Page submissions

Alternatively, individual page submissions can often be made directly to search engines via their respective interfaces. This manual method of page discovery can be used when new content is published on site, or if changes have taken place and you want to minimise the time that it takes for search engines to see the changed content.

Google states that for large URL volumes you should use XML sitemaps, but sometimes the manual submission method is convenient when submitting a handful of pages. It is also important to note that Google limits webmasters to 10 URL submissions per day.

Additionally, Google says that the response time for indexing is the same for sitemaps as individual submissions.

Comments

Post a Comment

Popular posts from this blog

Social Media Statistics In 2020

Image
  Have you planned a social media marketing strategy for your brand, yet? Like every year,  I have compiled a list of social media statistics that’ll help you draw an unbeatable social media marketing strategy for 2020. These social media marketing statistics will tell you -  How many social media posts are shared per day? What is the average time spent on Facebook or Instagram? What is the average engagement statistics of LinkedIn? Which is the best social media marketing platform for your brand? And much more... So whether you are heading a small/medium business, a part of the social media team of a company or a marketer looking to build his/her social media presence - these marketing insights will help you understand the changing dynamics of major social platforms. But before we get to that, let me tell you how I’ve divided the statistics into various sections: Demographic Statistics  - These numbers tell you about the characteristics of the population who are usi...
Read more

Outbound Link

  What Is an Outbound Link? An outbound link, also called an external link, is a link from your website to a  different  website. An outbound link for you is an inbound link for someone else. If a link takes readers to another page on the  same  website, it’s called an internal link. Controversies about outbound links abound, including these misconceptions: You should always avoid external links, because a) they send people away from your site, and b) you don’t have control over material on the other end of the link. The best way to keep people from leaving your site via outbound links is by opening them in a new browser window. Outbound links should always be tagged as nofollow so you don’t dilute the equity of your page. Not only are these myths, but following them might bring negative consequences. Why Are Outbound Links Important? Outbound links are one way that people and search engines discover useful content. They are important because they can ...
Read more

Google’s algorithm core update

Image
  Google’s algorithms   are a complex system used to retrieve data from its search index and instantly deliver the best possible results for a query. The search engine uses a combination of algorithms and numerous ranking signals to deliver webpages ranked by relevance on its search engine results pages (SERPs). Google makes thousands of changes every year. 2020 Core Update Google announced it is rolling out a new core update, the May 2020 core update, Monday. This is the second core update of the 2020 year, the first one was the January 2020 core update. Historically, Google has released a core Google algorithm update every few months or so. It was just a bit over 3-months ago since the January core update. The announcement.  Google said “Later today, we are releasing a broad core algorithm update, as we do several times per year. It is called the May 2020 Core Update. Our guidance about such updates remains as we’ve covered before.” Previous updates.  The most rece...
Read more