XML sitemaps and SEO Guide

What are Sitemaps?

As defined by sitemaps.org, organization that standardized the sitemaps protocol, it is one of the methods to inform search enigne about the pages in your site. There are many ways to do that and Sitemaps are one of them. Sitemaps can be in various formats like html, php or the most popular format xml. XML sitemaps are widely accepted standard by most of the websites and are used on small as well as very large websites with thousands of links listed in sitemaps. It lists urls of a site along with othre meta information like the date published and the importance of the page within the sitemap or website. It also contains the information about how frequently a page listed in the xml sitemap changes and how to inform about it to various search engines like Google, Yahoo, Bing, Ask, etc.

It is not guaranteed that using the sitemaps you get listed in all these search engines, it is just a method to remind them about the pages and whether they are updated/added or not.

Why Sitemaps standard is needed?

Sitemaps standard or protocol provide a basic communication link or channel between the search engines spider/crawler and the websites sitemap. Before the protocol html sitemaps were used and are not upto the mark as far as performance was concerned. Theoretically search engines need to use less processing power as they have the overview of the site available with them and can help to find them all the urls of a website with ease and risk of leaving the site before it is crawled wholely is reduced.

Sometimes important pages were left behind by the spiders and useless pages were indexed this was the biggest reason behind the stiemaps protocol first developed and accepted by google and than adopted by other search engines too.

XML Sitemaps Format:

Sitemap protocol is consisting of xml tags that are used as the basic structure of the sitemap file, and all the values (data) in the sitemap must be entity escaped and the sitemap file must be encoded in utf-8 format only. There are few other must haves that you need to keep in mind while writing your sitemap or creating a tool or plugin to create sitemaps automatically for CMS like joomla or drupal or blogging platform like wordpress. They are listed below as described on Sitemaps Protocol site:

1. A sitemap must begin with an opening <urlset> tag and end with a closing </urlset> tag.
2. You need to specify the namespace (protocol standard) within the <urlset> tag.
3. Must need to include a <url> entry for each URL, as a parent XML tag and include a <loc> child entry for each <url> parent tag.
4. All other tags are optional and support varies for different search engines like yahoo and google.
5. All the urls listed in a particular sitemap must be from a single domain like example.com as an example.

For more information about sitemap protocol and sample sitemaps and xml tags please visit this link at sitemaps.org

Sitemaps standard and Search Engines:

Google Sitemap Submit

In early 2007 all major search engines agreed to follow the sitemap xml standard adopted by google and created a website sitemaps.org to give more information about the standard and protocol used for the xml sitemaps. Yahoo is now accepting sitemaps in xml format along side its .txt format in yahoo site explorer. Bing also have started to accepting sitemaps in xml format after some initial testing for the format. Though sitemap can help find search engines hidden content but it should not be created and used for this sole purpose because it may send a wrong message to search engines and your site may get banned from search results too.

Before the sitemaps protocol was invented it was difficult to both webmasters and search engine spiders/crawlers to keep track and record of the various pages and links on the website but this protocol has made it very simple for site owners to create and update their sitemaps. There are various plugins available like Joomap for joomla and Google XML Sitemaps for WordPress self hosted blogs. This is a win-win situation for both webmasters and search engines as less resources are used to index the website and all the links from it with the help of this protocol.

Tools to create sitemaps:

There are various tools and scripts that you can use to create sitemaps. As described before joomla and wordpress can have their plugins installed and activated to create sitemaps and submit them to search engines automatically. All you need to do is to apply some settings in the plugin and save them and they are generated on the fly as links are generated on your blog or joomla website. There are other ways like some independent scripts or tools that you can use to create offline sitemaps on different operating systems like windows and linux.

GSitemaps is one such tool that you can use on any windows machine to get your website crawled and sitemap/s created automatically for you. Than all you need to do is to upload the generated xml sitemap to root of your website and inform search engines about its presence using webmasters tools, robot.txt or pinging services. There are few websites that offer this service for free for limited number of links and one such website is xml-sitemaps.com. You don’t need to download anything but the urls supported in free version are limited to few hundred links only.

PHPsitemapNG is a php script written for google sitemaps generation. GSiteCrawler is a windows based tool for the same. And then there are few web based tools like one java web based tool Sitemap Generator and Sitemap Builder. There are various other tools also available for sitemap generation and are extensively used in recent time for small and large websites alike.

For more information about this please visit the Wikipedia Article about sitemaps. You can read google’s FAQ about sitemaps or may visit the official blog of Sitemaps. You may also read Google Sitemaps Newsgroup for regular updates from the sitemaps protocol and standards.

Future and Developments in Sitemap Protocol Standard:

Initially there were two kind of sitemap submissions were available in google webmasters tools; web sitemap and mobile sitemap. Afterwards there were other formats added like Code Search Sitemaps, Geo Sitemaps and Video Sitemaps. So what is the use of these various formats and how they can be utilized. What is the future of this protocol in general? Protocol is new and implemented by search eninges since early 2007 and not much is changed since. So as we are moving towards the web 2.0 and standards are going to change as the complexities and traffic is going to increase in future.

Video on Google Webmasters Tools – Sitemap Submission::

Benefits of using a sitemap for your website:

It is not a replacement of your standard regular SEO or search engine optimization in anyway. It just can be used as an added tool to your online marketing efforts. It can be used for various purposes like indexing or blocking certain urls from search engines. There are various other SEO tricks and techniques related that we are going to discuss some other time. So what are the basic advantages?

• More efficient crawling by spiders as all information about pages to crawl are with them beforehand.
• No waiting for spider to come to your site as you can remind it using submit option most search engine webmaster tools provide now a days.
• Can inform search engines efficiently and almost in real time when content is updated on your website or comments are added to your blog.
• Can get almost all your urls indexed by major search engines without much effort by you so lot of man-hours are saved.

And an advice at the end of the story that don’t use this format to spam your urls as search engines now use efficient algorithms to detect spam in your index and can remove or altogether ban your website from appearing in search results. The standard is not going to change much and going to remain their for a long time as it is very new protocol so We recommend that you learn about it as much as possible.