Credits
Internal Linking Tool is built on open source work by other people. Every library it calls directly is listed here with its license.
Libraries in use
| Library | License | What it does here |
|---|---|---|
| crawlee | Apache-2.0 | the request queue and the crawl loop for a whole-site crawl |
| cheerio | MIT | reading links and titles out of the HTML a page serves |
| sitemapper | MIT | reading sitemap.xml, sitemap index files included |
| robots-parser | MIT | reading robots.txt rules, the Crawl-delay directive included |
| request-filtering-agent | MIT | refusing connections to private and reserved addresses, on every redirect hop |
| graphology | MIT | the internal link graph itself |
| graphology-metrics | MIT | PageRank over the link graph |
| graphology-traversal | MIT | click depth, by breadth-first search from the home page |
| natural | MIT | TF-IDF similarity over page text, which is what pairing pages by topic uses |
| @mozilla/readability | Apache-2.0 | splitting a page's main content from its template, so a link counts as contextual or not |
| linkedom | ISC | the DOM implementation the main-content reading runs on |
Apache-2.0 notices
Two of the libraries above are licensed under the Apache License, Version 2.0. Neither one ships a NOTICE file inside its published package, so the notice each one requires is reproduced here, in full.
crawlee
Copyright the crawlee contributors and Apify.
Published by Apify (apify.com), developed at crawlee.dev.
Licensed under the Apache License, Version 2.0. Source: https://github.com/apify/crawlee
@mozilla/readability
Copyright (c) 2010 Arc90 Inc
Developed by Mozilla and the Arc90 team.
Licensed under the Apache License, Version 2.0. Source: https://github.com/mozilla/readability
The full text of the Apache License, Version 2.0 is at apache.org/licenses/LICENSE-2.0. Each library is used as published: no file in any of them is modified, and the license text of each one ships inside its own package.
What is not here
No AGPL, GPL or SSPL licensed code is used anywhere in this product, and no hosted embedding service is called: the topic similarity is computed here, on this server. The complete dependency tree, with versions and licenses, is in pnpm-lock.yaml in the source repository. The same list is shipped as THIRD-PARTY-NOTICES.md.