InternalLinkingToolBot
How the Internal Linking Tool crawler behaves when it visits your site, and what to do if you would rather it did not.
User agent
Every crawler request identifies itself with this user agent string:
InternalLinkingToolBot/1.0 (+https://internallinkingtool.com/bot)A whole site is crawled only after its owner has verified it, and the verification is re-checked before every crawl, so we do not turn up on a site that has not asked for us.
robots.txt
The crawler obeys robots.txt. It reads your rules for the user agent token InternalLinkingToolBot, follows the sitemap the file points at, and drops any URL your rules disallow. A Crawl-delay directive is honored as well, up to 10 seconds, so a site can slow us down without stalling its own crawl indefinitely.
A single-URL check you run yourself fetches only the page you give it, and it does not consult robots.txt, because it is a request you made about a page you typed in. Whole-site crawling is the part that reads your rules.
Pace
- One request at a time per host. We never run parallel requests.
- At least 500 ms between two requests to the same host, and longer when your Crawl-delay asks for it.
- 1.5 seconds between requests on a Shopify host, which identifies itself in its response headers and is slower to serve.
- One crawl per host at a time across all accounts, so two customers cannot both be crawling your site.
Backoff and pausing
If your server answers 429, 430, 503, we treat that as a request to slow down. We wait for the Retry-After header when it is there, and otherwise wait 30 seconds, 60 seconds, 2 minutes, 4 minutes on each consecutive attempt. No single wait is longer than 15 minutes.
After 6 of those in a row the crawl pauses instead of failing, and resumes on its own. If the site keeps rate limiting us through several pauses, the crawl stops and the account owner is told that the site kept rate limiting us, rather than being left with a crawl that looks stuck.
What it never does
- It never bypasses bot protection. A challenge page is an answer: we stop and report that the site blocked us.
- It never runs a browser, so no script on your pages is executed and nothing on your page is clicked.
- It never sends a request to a private or reserved network address, and it uses the standard http and https ports only.
How to block it
Add this to your robots.txt:
User-agent: InternalLinkingToolBot
Disallow: /You can also block the user agent string at your server or firewall. If your site answers a crawl with a bot protection challenge, we take that as a no: the crawl stops and reports that the site blocked us.
Contact
A question about a crawl we ran on your site, a block we should honour, or a request to delete data: write to [email protected]. The privacy page says exactly what is stored and for how long.