View your website like search engine spider would!

Screaming Frog - View Your Website Like A Search Engine Spider

By Lee Kane


Ever wanted your own personal search engine spider, that could simulate what a spider see’s as it goes crawling through your website looking for information to send back to its search engine. Well you can and its free upto 500 URI (Uniform Resource Identifier).

search engine spider

Note! Which means it will check upto 500 resources including URL’s (which technically, so a gathered, are also URI). Why is that important! Screaming Frog might not return all the URI associated with a URL, so don't be to worried if the free version does not display certain pages or resources. You will have to buy the license to have this limit removed.

Screaming Frog is your own personal search engine spider and although the paid for version (£99 per annum) offers far more options for analysis, the free version will give you lots of pointers.

screaming frog logoScreaming Frog is software, it has to be installed on your PC after which you can you crawl your own website and your competitors, enabling you to compare your website to theirs. It gives you the ability to see all in one place, the page titles on your site, meta descriptions, meta keywords and the character lengths of all of them, giving you the opportunity to review where you can make improvements.

For instance when looking at the data on my website you can see where i'm not making the most of the page titles by toggling the column to display the shortest.

screaming frog screenshot titles
screaming frog screenshot description

Another feature is its ability to display the HTTP response status codes. When i first crawled my website with Screaming Frog, i noticed all the 404 error codes returned by websites that i had linked too from my website (broken links). 404 error codes have a real effect on SEO, as it gives of a signal to search engines that the website is not being maintained, so being able to see exactly which links where returning this code was a great help in getting them fixed fast.

screaming frog screenshot 404

The response status code feature also helps with spotting the not so obvious problems within your website. There were a couple of strange 404’s returned on pages that didn't exist on my website, but knowing that this problem was there enabled me to apply 301 redirects to them using the htaccess> file and put a temporary fix to a problem that i had no idea existed.

As with the titles, keywords and descriptions, Screaming Frog also displays the H1 H2 heading tags on your website pages along with their content and lengths. Again being able to analyze and review your own headings and compare them with competition websites is just one of the many valuable assets of this tool.

With images Screaming Frog can display the URI of your images and their sizes, this enables you to see which images on your website are the largest, making it easier for you to spot the images which need compressing.

screaming frog image search
screaming frog image sizes

All the data returned from crawling your own, or your competitor's websites, can be viewed together or individually, crawls can also be customized just to return the data you want, and all data can be downloaded as a CSV file, which helped me in creating spreadsheets of the content i had sharded for parallelization.

These where just the main areas i was able to put this tool to work for me and it has far more to offer as you can read by following the links below. Screaming Frog is worth having in your array of SEO tools.

8 Alternative Ways To Use Screaming Frog for SEO

Screaming Frog Guide to Doing Almost Anything: 55+ Ways of Looking at a Tool

*All logos and trademarks used are registered trademarks of their respective owners.

You Might Also Like

Share your thoughts!