Home -> My Blog -> SEO Article Blog -> Search Engine Optimization Crawling

Search Engine Optimization Crawling

In order to understand Search Engine Optimization we need to understand crawling first.A webpage creator wants his or her webpage to receive a high ranking that will make the webpage easily accessible and visible to searchers. This is, after all, what search engine optimization is all about and why optimizers go through the trouble of analyzing competitors, researching ranking factors and updating factors on their own website. However, time and effort of analyzing and updating a webpage is lost if the optimizer is unable to understand the method by which their webpage receives a ranking.

Search engine crawling is how each individual webpage receives a ranking.Search-Engine-Optimization

Search engines all have their own unique algorithms that are used to rank webpages—but these algorithms don’t simply exist, floating about the internet! Algorithms must be applied to web pages and the process of application is what is generally referred to as search engine crawling.

Search engines have a continuous system: crawl, index, rank and repeat. They search the internet, examining webpages, indexing content (organizing URLS, labeling content etc), and assigning ranking. Google’s search engine “spider”, called Googlebot, is continuously searching the internet, indexing webpages, assigning and updating ranks.

The system is complex and fascinating, and something that webpage creators and optimizers take for granted if they do not realize one important fact: search engine spiders and crawlers favor some webpage layouts over others.

While accessibility to search engine crawling is often considered one of the general factors that are used to determine ranking in search results, if search engines are not able to easily crawl a webpage, many positive factors of the webpage (Ex. keywords, content, and links) may be overlooked. An optimizer may continuously update a webpage to increase ranking without seeing any results if he does not take the search engine crawler into account.

Websites are more than colored blocks, pictures and text; they are codes and creations that are all vastly different than one another. Interactive websites can be made using and incorporating many different methods and elements, including HTML, Flash, PDF, ActionScript, and JavaScript.

It is important for an optimizer to understand the building blocks of his or her website. For example, it has been universally accepted that Flash is not particularly adept at handling text (at least when compared to basic HTML). Flash does not contain the same setup of text and links that are present in HTML websites. This often creates nothing more than minor loading problems, but was once catastrophic when SEO was brought into the equation. There are many web crawlers that have trouble reading websites created with Flash—in the past, as much as 40% of the webpage content would go unrepresented in the examination and subsequent assignation of ranking because of web crawler’s (Googlebot, for example) inability to read webpage content. This has been changed over time since the advent of Flash, but Flash, along with other creation platforms and scripting systems, is still subject to flaws.

Other platforms also encounter problems with search engine crawlers. Javascript often goes almost entirely unread by crawlers. Links of PDF files within a webpage often go unnoticed by web crawlers as well. Websites that utilize a doorway page/splash page are also often misread by search engine crawlers.

However, search engine crawlers are still evolving and learning.

It is the job of the optimizer to create a webpage that is easily readable by crawlers. Crawlers are what give webpages a ranking, so it’s better to be friendly to them rather than make things difficult. There are many tips and tricks that an optimizer can utilize to make a webpage layout more crawler-friendly.

A simple online search can provide lists of tips to try out, but some of the more popular ones include:

· Embed less accessible Flash, Javascript, etc. into HTML pages

· Don’t use less accessible formats for navigation

· Save less accessible formats for design elements

· Include descriptive titles for web page

· Include descriptions of links

In the end, it is up to the optimizer and creator to research the crawler of the search engine they wish to optimize with and play to the necessities of the crawler. Sometimes elements of a webpage may by informative and, overall, quite positive for the ranking of the webpage, but if they are not formatted to provide easy access for the crawlers, all of the formatting, research, and creation work is for nothing.

Share This