A version of this post originally appeared in the October 22, 2020 issue with the email subject line "When 404s and 301s swarm" and an essay about managing 404 and 301 zombie links.
When recommending tech for clients, my number one rule is: Don’t change what’s already working. Keep what’s already doing its job well.
Software companies and SEO agencies are always trying to make a better, sexier web crawler than Screaming Frog, which compared to all the pretty new tools out there, looks more like a DOS screen every year. But good old SF does its job, quickly, with few frills. It’s utilitarian, it’s worth every penny of the low license fee, and I know how to use it.
Screaming Frog is free for basic crawls up to 500 pages, and for most small websites, that’s all you need. An annual license for the unlimited full-featured version is under $200 in the current exchange rate.
Screaming Frog at a glance
In addition to identifying html status errors like 404s and 301s, Screaming Frog can illuminate a number of technical SEO issues, including:
- Duplicate content identification
- Missing titles, meta descriptions, and header tags
- Overly long or incorrectly implemented page and meta content
- Canonicalization
- Site structure
- Structured data/schema markup
- Google snippet demo visualization
The tool generates high-quality sitemaps, links with Google Analytics and Search Console for individual page analysis, crawls Javascript websites that are harder to read than standard HTML, and generally supplies everything you need to audit a website’s health.
You can find oodles of resources to help you with Screaming Frog analysis. I like SF’s tutorials and this guide from Seer Interactive.
It’s mostly a professional-grade SEO or content auditing tool, but it’s worth learning if you want to get into the nitty gritty of website maintenance.