Short guide on submitting your web site to the popular search engines
When you first set up your web site, you must feel very excited to announce it to the world. You can ask your friends to check your web site and have them to add links to your web site from theirs if they have any. You can also submit your web site directly to the popular search engines so they can start crawling your web site and add its content to their index. Following we describe how to do this with Google, Yahoo!, Microsoft Live search and Ask.com.
Although it is possible to submit your web site's URLs to these search engines anonymously, it is much better for you to register accounts at these sites so you use their tools for webmasters. For example, you can track how they crawl your web site and whether there are any issues with your pages. Common sense tells us that anonymous submission gets little credibility to your web site and it may get very low priority in processing. Besides you don't know what and when will happen next.
1. Submit your site to Google
You can submit your web site through Google webmaster tools by visiting Google Webmaster Central. You first need to create an account with Google and then you can sign in to submit your web site. From the dashboard, you can check whether your web site has been added to the search index or not.
1.1 Site validation with Google
Earlier we said anonymous submission gives you little credibility.
Now you need to get your site validated so you can track various
statistics. The usual way from search engines is to ask you to upload a
specific file to your web site, such as google123456789abcdef.html.
Because usually only you can do this, this is a very effective way
for validation. You should also put the file in your version
management system so you won't lose it.
1.2 Sitemap generation for Google
In addition to submitting your web site, you should also generate a sitemap for Web crawlers. In a sitemap, you can inform them what pages to crawl, how often pages are updated and etc. All the major search engines support the Sitemaps protocol.
First you need to create or modify the robots.txt
under your web site's root directory. You just need to add
one line like the following:
Sitemap: http://www.example.com/sitemap.xml
Then you need to create the sitemap.xml file.
You can use
this site's sitemap.xml as a starting point to
work on your own sitemap.xml file. If you use
HTML::Mason, the powerful Perl based template management
system, to manage your pages, you can also use the
sitemap.xml sample template file in Mason to see how
to generate the sitemap.xml file programmatically.
Whenever your web site's sitemap file changes, you can wait for
the crawlers to discover it by themselves or you can resubmit the file
manually through the Google Webmaster Tools or make an HTTP request
to Google's ping service like this:
http://www.google.com/webmasters/tools/ping?sitemap=http%3A%2F%2Fwww.example.com%2Fsitemap.xml
The latter is mostly used by automation script that makes the request
to Google whenever it detects your site changes. Please note here
the URL for your site's sitemap needs to be URL encoded properly.
2. Submit your site to Yahoo!
Yahoo! Site Explorer
allows you to submit your web site. As we have said earlier,
you can create a free account at Yahoo! so you can track your site's
various statistics. For validation, it also requires you to
upload a file to your web site's root directory like this
y_key_abcdef123456.html.
You also need to keep this file around because continuous validation
may be required.
To submit your web site's sitemap to Yahoo!, you need to use
the
Site Explorer Web Services ping API which does not require any
authentication.
The usage looks like this:
http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=http%3A%2F%2Fwww.example.com%2Fsitemap.xml
You can integrate it into your automation script to notify Yahoo!
whenever you web site changes.
3. Submit your site to Microsoft Live search
Although you can submit your site anonymously through the Live search URL submission page, we still suggest you create an account and then do it through the Microsoft Live Search Webmaster Center.
There are two forms of validation. One is to upload an XML file, the other
is to add a meta tag to your home page file. We don't have much luck with
the second one because the
W3C Markup Validation Service does not like the resulting page and
Live search validation also fails. So we suggest you upload an XML file named
LiveSearchSiteAuth.xml and put the right user ID there
for validation. You can check
this site's LiveSearchSiteAuth.xml for an example.
You can also submit your site's sitemap.xml file through the Webmaster
center. Whenever it changes, you can notify Live search by
sending a request to the following URL
http://webmaster.live.com/webmaster/ping.aspx?siteMap=http%3A%2F%2Fwww.example.com%2Fsitemap.xml.
Please remember to substitute the right sitemap URL.
4. Submit your site to Ask.com
According to the
Ask Website Crawler FAQ, you can submit your sitemap file
through a URL like the following
http://submissions.ask.com/ping?sitemap=http%3A%2F%2Fwww.example.com%2Fsitemap.xml.
5. Perl script to submit your site's sitemap file
For your convenience we provide a sample Perl script to submit sitemap's URL to the above four sites that you can adapt to your own.