Become a Googlebot, get a Free Pass

by Kaushik November 18, 2006

The Problem:

Sometimes you conduct some search on Google and the engine returns a number of results, but when you try to open the ones that looks the most promising you get a registration page. One thing that you can do is click on the cached content to view the page directly from Google's cache. But what if you want to view another page from the same site, not present in google's cache?

The Solution:

Its evident that there are some sites that allows Google but not you. So why not disguise yourself as Google? Thats exactly what we will do by changing the browser's user agent to Googlebot. Copy the following code into notepad and save it as a .reg file









Now double-click the file to merge it into registry. In order to change back to the default user agent use this:


Unfortunately, this only works with IE and not Opera. There is no way to add new user agent in Opera AFAIK. There is some way to do it in Firefox but you have to search that yourself on google. To check whether you have sucessfully changed the user agent, type the follwing into the address bar and hit enter.


Now for some practice. One good example is http://www.governmentsecurity.org/forum/index.php. Anybody who has tried to visit this website will find that it requires you to register in order to view the forum. But registering is tricky. You need to submit your own article to the site afterwhich they will evaluate it and then decide whether to give you membership or not. Now visit this forum after spoofing your IE's user agent, aaha you are welcomed!!!

Note, that this method might not work with all websites.

Fun Exercise 1:






Use a different browser like Opera or Firefox to login to thinkdigit.com. Then click on Quick Links>Who's online (you will also find this near the bottom of the page). On the page that opens choose "Search Bots" and hit go. No matter what time of the day or night it is, you will find a couple of googlebots, msnbots and yahoobots crawling on the website. Now open IE and view thinkdigit.com, this time as a googlebot. Refresh the page you were viewing in your other browser. This time you will see another googlebot appearing which is actually you!!

Fun Exercise 2:

On the same "Who's online" page you can see what thread the other online members and guest are viewing at that particular time. Speaking of privacy, huh?

Update: (28th April,2k8)
_____________________________________________________________________________
Here is a plugin newbies can use to work. This is just a plugin for firefox(refered by madcadder below in his comment).
Steps:
1)Just install the plugin(Configuration Mania) in firefox from the site url given below.
2)Once done restart your firefox browser.
3)Go to Tools --> Configuration Mania
4)Click on UserAgent --> GoogleBot
Thats it,now you can run those examples given above.. No need of doing any registry changes.

Here is one more example:
Click on the link below before switching to googleBot:
Example Link
It will ask you for userId and password.
Now switch your browser using this plugin and you dont need any userId or password any more.. Yahoo!
Link: Plugin Link
_____________________________________________________________________________