IP Addresses of Search Engine Spiders

Are you interested in tracking the IP addresses of search engine spiders? You can see my latest lists of search engine IP addresses and User-Agents here:

Google IP List
List of the IP addresses of the various Googlebots and Mediabots.
Last revised Tuesday, 20-May-2008 12:57:35 CDT Size: 4.0K

Yahoo IP List
List of the IP addresses of Yahoo/Inktomi bots.
Last revised Thursday, 21-Aug-2008 23:34:51 CDT Size: 11K

Lycos IP List
List of the IP addresses of Lycos bots.
Last revised Tuesday, 20-May-2008 12:57:37 CDT Size: 580

InfoSeek IP List
List of the IP addresses of InfoSeek bots.
Last revised Tuesday, 20-May-2008 12:57:38 CDT Size: 1.3K

Alta Vista IP List
List of the IP addresses of Alta Vista bots.
Last revised Tuesday, 20-May-2008 12:57:36 CDT Size: 1.9K

Excite IP List
List of the IP addresses of Excite bots.
Last revised Thursday, 22-May-2008 10:37:58 CDT Size: 291

Northern Light IP List
List of the IP addresses of Northern Light bots.
Last revised Tuesday, 20-May-2008 12:57:38 CDT Size: 211

Misc IP List
List of the IP addresses of various search engines' bots.
Last revised Thursday, 21-Aug-2008 23:34:45 CDT Size: 61K

Non Spiders
List of IP addresses and some User Agents of non-search engine spider bots, along with some other pertinent information.
Last revised Tuesday, 20-May-2008 12:57:38 CDT Size: 6.5K

Here is a record of updates to the IP lists.

A reorganized list is available here.

Notes

Tracing search engine IP addresses is handy for a number of reasons. You might want to see when your web site was spidered by search engines. You might want to know which pages where spidered. You might want to ban some bots and let others in. You might want to see which bots are obeying your robots.txt files. You might want to redirect bots to other pages or serve them different content (cloaking).

Keeping track of the various IP addresses belonging to search engine spiders is a difficult enterprise and requires the assistance of a large number of people. I'm constantly receiving reports of new spiders and doing the research necessary to verify that they are indeed new and that they do indeed belong to search engines. The lists are as up to date as possible, but there is no guarantee that they include all of the IP addresses of all search engines (it would be impossible to guarantee such a thing). Also, there is no guarantee that all of the IP addresses contained in the lists belong to search engines (they are sometimes reassigned or abandoned).

I do accept submissions of suspected bot activity. If you see some bot activity on your web site and want to submit it, please do the following:

  1. Record the line from your server's access log.
  2. Email me with the line from your server's access log.
  3. Please don't send email if it isn't about a new bot or directly related to this site. I can't track down the owners of IP addresses or figure out where spam is coming from... but I have written a good tutorial on how to track down spammers.

The IP addresses are contained in a flatfile database format, which is protected under intellectual property law and treaties, however the IP addresses themselves are public domain and not protected. What this means to you is that you can't copy this website, but feel free to download the lists of IP addresses and use them however you like.

If you are writing an application to download the IP addresses automatically, please ask me for permission first.

FYI -- your IP address is: 38.103.63.60 .

If this site has been helpful to you, you might consider buying me a drink! ;)

Resources

This website is (C) Copyright 2001 by Volatile Graphix, Inc. The IP lists themselves are public domain and not subject to copyright protection.