By 2004, search engines had incorporated a wide range of undisclosed factors in their ranking algorithms to reduce the impact of link manipulation. In June 2007, The New York Times' Saul Hansell stated Google ranks sites using more than 200 different signals. The leading search engines, Google, Bing, and Yahoo, do not disclose the algorithms they use to rank pages. Some SEO practitioners have studied different approaches to search engine optimization, and have shared their personal opinions. Patents related to search engines can provide information to better understand search engines. In 2005, Google began personalizing search results for each user. Depending on their history of previous searches, Google crafted results for logged in users.
To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots. When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.
For a long time, digital marketers summed up the properties of direct and organic traffic pretty similarly and simply. To most, organic traffic consists of visits from search engines, while direct traffic is made up of visits from people entering your company URL into their browser. This explanation, however, is too simplified and leaves most digital marketers short-handed when it comes to completely understanding and gaining insights from web traffic, especially organic and direct sources.
Website saturation and popularity, or how much presence a website has on search engines, can be analyzed through the number of pages of the site that are indexed by search engines (saturation) and how many backlinks the site has (popularity). It requires pages to contain keywords people are looking for and ensure that they rank high enough in search engine rankings. Most search engines include some form of link popularity in their ranking algorithms. The following are major tools measuring various aspects of saturation and link popularity: Link Popularity, Top 10 Google Analysis, and Marketleap's Link Popularity and Search Engine Saturation.
This all sounds amazing right? unfortunately, organic marketing is also very difficult to implement and resource intensive. Ranking organically on Google and other search engines can be very hard, especially for competitive keywords such as “buy shoes.” Ultimately you want to rank within the first page, and possibly within the top 3 organic results, but this can be next to impossible when you are competing with large companies that have teams dedicated to ranking their keywords.
BrightEdge research supports that a blended approach is best for delivering high performing content. Not only will combining organic and paid search increase website traffic, but it will offer a bigger return on the investment. Take Retail, Technology and Hospitality industries, for example — organic and paid search combined make up more than two-thirds of their total revenue.