Crawling notice

AI2 Bot

A bot, also known as a web robot, web spider, or web crawler, is a software application that runs automated tasks over the Internet in a more effective, structured, and concise manner than a human could ever do.

The AI2 Bot explores certain domains to find web content. This web content is used to train open language models. If you have any questions or concerns about our crawler, please contact us.

Technical Details

Our crawler always makes requests with the following User-Agent HTTP header when looking for documents:

Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)

This user agent string can be used to filter or reject traffic from our crawler if desired.