Pra quem já se perguntou como o web.archive funciona, o crawler usado por ele é opensource e é esse:
https://github.com/internetarchive/heritrix3
Doc: https://heritrix.readthedocs.io/en/latest/
https://github.com/internetarchive/heritrix3
Doc: https://heritrix.readthedocs.io/en/latest/
GitHub
GitHub - internetarchive/heritrix3: Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web…
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. - GitHub - internetarchive/heritrix3: Heritrix is the Internet Archive'...
