How to Use Site Crawler

What is Site Crawler?

Site Crawler is a simple yet powerful tool that allows you to crawl any website, be it your own website, your client's website or even your competitors' website. This tool helps you analyze a website's internal link profile, to discover which pages link where, uncover dead links and HTTP errors, view the anchor profile and much more.

How to Use Site Crawler?

Go to Site Crawler, which is located under the Site Management section, in the TOOLS menu.



To start a new crawl, click the blue + button located at the top right of the page and input the website address, specify the number of links that you want to crawl and click Crawl.



Note that it costs 1 credit per link crawled. The number of links that you specify in the form is the maximum number of links that will be crawled. This is to prevent infinite looping from happening due to large number of links and to ensure that you have control over the number of credits you want to spend on a crawl. For example, if you specify 1,000 links and the website only has 900 links, then once the crawl is completed, 100 credits will be credited back to your account. The number of credits available to you is dependent on your subscription package.

The tool will start to crawl your website. You will receive an internal notification message once the crawl in complete.

Once completed, you will see the crawl results in the history list. Here you can see details such as the number of pages crawled and number of links found. Click on the 3-dot icon on the right hand side of the crawl results to View, Re-Run or Delete the results.



From the crawl results page, you will see a list of the links crawled with the following information:

URL of the links crawled
Link status e.g. OK, Client Error, Redirection and so on...
HTTP Status Code e.g. 200, 404, 301 and so on... You can click on the arrow next to each code to get an explanation of the code.



Click on the magnifying glass icon on the right hand side of each URL, to get a detailed breakdown of the links that links to and links from this website, including whether they are DoFollow or NoFollow links and SiteWide or not SiteWide links, i.e. on all pages of the website or not.



Below the list of links, you will also get an Anchor Text Breakdown, in the form of a pie chart.

Updated on: 21/06/2021