Crawlers crawl billions of pages. To do this, they follow pathways formed primarily by internal links - these links must…