Web Crawlers Detection in ASP.Net

Tuesday, April 20, 2010 2:18 AM Posted by Hrishi
Labels: ,


Here is a way of detecting search engine (Google, live…) crawlers,spiders etc..


 System.Web.HttpBrowserCapabilities clientBrowserCaps = Request.Browser;
            if (((System.Web.Configuration.HttpCapabilitiesBase)clientBrowserCaps).Crawler)
            {
                Response.Write ( "Browser is a search engine.");
            }
            else
            {
               Response.Write ("Browser is not a search engine.");
            }

            //Here is another approach..

            if (Request.ServerVariables["HTTP_USER_AGENT"].Contains("Googlebot"))
            {
                //log Google bot visit
            }
            else if (Request.ServerVariables["HTTP_USER_AGENT"].Contains("msnbot"))
            {
                //log MSN bot visit
            }
            else if (Request.ServerVariables["HTTP_USER_AGENT"].Contains("Yahoo"))
            {
                //log yahoo bot visit
            }
            else
            {
                //similarly you can check for other search engines
            }

Comments (0)

Post a Comment

Subscribe to: Post Comments (Atom)